Oct 29 22:39:49.825510 ip-10-0-134-205 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 29 22:39:49.825524 ip-10-0-134-205 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 29 22:39:49.825533 ip-10-0-134-205 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 29 22:39:49.825866 ip-10-0-134-205 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 29 22:40:00.056291 ip-10-0-134-205 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 29 22:40:00.056309 ip-10-0-134-205 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 995fd839fb92401698f31d22a369a05f --
Oct 29 22:42:04.079030 ip-10-0-134-205 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:42:04.527022 ip-10-0-134-205 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:04.527022 ip-10-0-134-205 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:42:04.527022 ip-10-0-134-205 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:04.527022 ip-10-0-134-205 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:04.530999 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.530919    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.530999 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.530998    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.530999 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531002    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531005    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531008    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531011    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531014    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531016    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531019    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531022    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531025    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531028    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531031    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531034    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531037    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531039    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531042    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531045    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531048    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531050    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531053    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.531089 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531055    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531058    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531061    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531064    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531067    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531070    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531078    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531081    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531084    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531087    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531089    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531092    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531095    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531098    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531101    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531104    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531106    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531109    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531112    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531114    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.531530 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531117    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531120    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531123    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531125    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531128    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531130    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531135    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531138    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531141    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531144    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531147    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531149    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531152    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531155    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531160    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531163    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531165    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531168    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531171    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531174    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.532007 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531176    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531179    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531182    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531184    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531188    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531190    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531193    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531196    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531199    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531202    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531205    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531208    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531210    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531213    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531216    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531219    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531221    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531224    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531227    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531230    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.532470 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531233    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531235    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531238    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531241    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531243    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531246    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531250    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531253    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531256    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531260    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531265    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531267    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531271    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531276    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531280    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531283    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531286    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531289    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531293    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.532956 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531296    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531299    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531301    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531304    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531307    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.531309    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532784    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532793    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532797    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532800    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532802    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532805    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532808    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532811    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532814    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532817    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532819    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532822    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532824    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.533397 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532828    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532831    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532834    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532837    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532839    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532842    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532845    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532847    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532850    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532853    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532855    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532858    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532860    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532863    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532866    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532869    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532871    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532874    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532877    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.533873 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532879    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532882    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532885    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532887    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532890    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532893    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532896    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532898    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532902    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532904    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532907    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532909    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532912    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532915    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532917    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532920    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532923    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532925    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532928    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532931    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.534333 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532933    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532937    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532939    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532942    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532945    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532947    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532951    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532953    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532956    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532959    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532962    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532965    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532968    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532971    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532974    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532978    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532982    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532985    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532988    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.534825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532991    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532995    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.532999    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533002    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533005    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533007    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533011    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533014    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533017    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533019    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533022    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533025    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533028    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533030    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533033    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533036    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533039    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533041    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533044    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.535300 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533047    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533051    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533053    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533056    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533059    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533062    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533064    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533067    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533069    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533072    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533074    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533077    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533079    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533082    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533085    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533087    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533634    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533640    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533643    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533646    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.535753 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533649    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533652    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533654    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533657    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533660    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533662    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533665    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533667    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533670    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533673    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533675    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533678    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533680    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533683    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533686    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533688    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533691    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533694    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533699    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.536316 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533703    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533706    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533710    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533713    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533715    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533719    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533722    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533724    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533728    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533731    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533734    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533737    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533740    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533743    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533746    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533749    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533752    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533754    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533757    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533760    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.536818 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533776    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533779    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533782    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533785    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533787    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533790    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533793    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533795    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533798    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533801    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533804    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533806    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533809    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533812    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533814    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533817    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533819    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533822    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533825    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.537301 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533827    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533830    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533833    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533836    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533839    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533842    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533845    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533847    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533851    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533853    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533856    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533859    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533861    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533864    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533866    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533869    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533872    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533874    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533877    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533880    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.537755 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533882    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533885    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533889    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533891    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533894    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533897    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533899    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533903    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533906    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533909    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533912    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533915    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533918    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533920    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533923    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533926    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533929    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533932    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533934    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.538290 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533938    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533940    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533943    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533945    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.533948    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535012    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535022    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535030    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535035    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535039    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535043    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535048    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535052    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535056    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535060    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535064    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535067    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535072    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535076    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535079    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535082    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535085    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535088    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535092    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:42:04.538727 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535095    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535098    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535101    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535105    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535109    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535113    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535116    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535120    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535123    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535126    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535130    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535133    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535136    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535158    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535162    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535166    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535169    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535172    2574 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535175    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535181    2574 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535184    2574 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535187    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535191    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535194    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:42:04.539322 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535199    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535203    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535207    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535210    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535213    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535216    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535219    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535222    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535225    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535228    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535231    2574 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535236    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535239    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535242    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535245    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535252    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535255    2574 flags.go:64] FLAG: --help="false"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535259    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535262    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535265    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535269    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535272    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535276    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535279    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:42:04.539928 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535282    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535285    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535288    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535291    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535295    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535298    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535301    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535304    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535307    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535312    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535315    2574 flags.go:64] FLAG: --lock-file=""
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535318    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535321    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535324    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535329    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535332    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535335    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535338    2574 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535341    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535345    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535348    2574 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535351    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535356    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535360    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535364    2574 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:42:04.540512 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535367    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535371    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535374    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535377    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535380    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535383    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535386    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535406    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535410    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535413    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535416    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535420    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535423    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535426    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535429    2574 flags.go:64] FLAG: --port="10250"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535432    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535436    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c3d01b4618fa85d2"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535440    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535443    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535447    2574 flags.go:64] FLAG: --register-node="true"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535450    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535454    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535457    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535460    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535463    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535467    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:42:04.541118 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535470    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535473    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535476    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535479    2574 flags.go:64] FLAG: --runonce="false"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535484    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535487    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535491    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535494    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535497    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535500    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535503    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535507    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535509    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535512    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535515    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535518    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535521    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535524    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535528    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535533    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535536    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535539    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535544    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535547    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535550    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:42:04.542005 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535553    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535556    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535559    2574 flags.go:64] FLAG: --v="2"
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535563    2574 flags.go:64] FLAG: --version="false"
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535567    2574 flags.go:64] FLAG: --vmodule=""
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535572    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.535575    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535734    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535739    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535742    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535746    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535751    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535755    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535758    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535761    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535775    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535778    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535781    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535784    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535787    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535789    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.543060 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535792    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535795    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535797    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535800    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535803    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535805    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535808    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535812    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535817    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535821    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535824    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535827    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535830    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535833    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535836    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535839    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535841    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535844    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535847    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535849    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.543972 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535852    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535854    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535858    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535861    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535864    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535867    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535870    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535873    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535876    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535879    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535881    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535884    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535887    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535890    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535893    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535895    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535898    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535901    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535903    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.544816 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535906    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535909    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535912    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535914    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535917    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535920    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535922    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535925    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535927    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535930    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535933    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535935    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535938    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535940    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535943    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535946    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535949    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535951    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535954    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535957    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.545636 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535960    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535963    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535965    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535968    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535970    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535973    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535977    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535980    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535983    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535986    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535989    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535991    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535994    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.535997    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536000    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536002    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536005    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536007    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536010    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.546543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536013    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536015    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536018    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536020    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536023    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536025    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536028    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536030    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536034    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536037    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536039    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536042    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536045    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536048    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.536075    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536113    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.547168 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536117    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536121    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536124    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536126    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536129    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536131    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536136    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536139    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536142    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536145    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536148    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536152    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536155    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536158    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536160    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536163    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536165    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536168    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536171    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.547700 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536173    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536176    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536178    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536182    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536184    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536188    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536191    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536193    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536196    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536199    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536201    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536204    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536207    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536210    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536212    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536215    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536217    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536220    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536222    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.548487 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536225    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536228    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536230    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536233    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536236    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536240    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536243    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536245    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536248    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536251    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536254    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536256    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536259    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536262    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536264    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536267    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536271    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536275    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536279    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536282    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.549191 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536285    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536288    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536291    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536294    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536296    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536299    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536302    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536305    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536308    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536310    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536313    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536315    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536318    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536321    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536324    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536327    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536331    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536335    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536338    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.549825 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536341    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536343    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536347    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536349    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536352    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536355    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536357    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536360    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536363    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536365    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536368    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536373    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536376    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536378    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536381    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536383    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536386    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536389    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536391    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536394    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.550312 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536397    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536399    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536402    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536404    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536407    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536410    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536412    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.536415    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.544559    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.544581    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.544605    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.544617    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544707    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544716    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544722    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544726    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544731    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544735    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544740    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544745    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.550854 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544750    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544754    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544758    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544778    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544783    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544787    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544792    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544796    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544800    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544804    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544809    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544812    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544817    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544821    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544825    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544829    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544833    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544838    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544841    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544845    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.551354 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544849    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544863    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544868    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544873    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544877    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544881    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544885    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544889    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544894    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544898    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544903    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544907    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544912    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544916    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544921    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544925    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544930    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544934    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544938    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.551954 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544942    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544947    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544951    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544955    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544959    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544964    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544968    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544974    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544980    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544985    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544990    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544994    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.544999    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545003    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545008    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545020    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545024    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545028    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545033    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.552437 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545037    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545041    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545045    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545049    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545054    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545058    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545062    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545066    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545070    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545075    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545079    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545083    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545087    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545091    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545095    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545099    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545103    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545108    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545112    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545116    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.552892 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545120    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545124    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545129    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545133    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545137    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545142    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545146    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545151    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545155    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545167    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545171    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545176    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545180    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545187    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545194    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545200    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545204    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545210    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545215    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.553543 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545219    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.545260    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545544    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545552    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545558    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545563    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545567    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545572    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545576    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545582    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545588    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545592    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545597    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545601    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545605    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545609    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.554232 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545614    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545618    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545623    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545628    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545632    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545636    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545640    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545652    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545656    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545660    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545665    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545669    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545673    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545678    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545682    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545686    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545690    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545695    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545699    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.554638 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545703    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545708    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545712    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545717    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545721    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545725    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545729    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545734    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545738    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545742    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545746    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545750    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545755    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545759    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545779    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545783    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545787    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545791    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545795    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545800    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.555160 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545804    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545819    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545824    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545829    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545833    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545837    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545842    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545846    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545850    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545855    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545859    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545864    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545868    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545873    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545877    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545882    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545886    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545890    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545894    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.555641 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545899    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545903    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545907    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545911    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545924    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545929    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545933    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545937    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545941    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545946    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545950    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545954    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545959    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545963    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545967    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545979    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545984    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545988    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545992    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.556114 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.545997    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546001    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546005    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546009    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546014    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546018    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546022    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546029    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546033    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546037    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546042    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546045    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546050    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546054    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546058    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.546100    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:04.556557 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546168    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546175    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546181    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546186    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546192    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546199    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546204    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546208    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546213    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546218    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546222    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546226    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546230    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546248    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546253    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546257    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546261    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546266    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546270    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:04.556951 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546274    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546278    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546283    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546287    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546291    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546298    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546302    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546306    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546310    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546314    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546319    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546323    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546327    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546331    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546335    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546340    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546344    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546348    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546352    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:04.557415 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546357    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546363    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546367    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546371    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546375    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546379    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546383    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546388    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546400    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546404    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546408    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546412    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546417    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546421    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546425    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546430    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546434    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546438    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546444    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546448    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:04.558024 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546453    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546457    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546461    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546465    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546469    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546476    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546481    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546486    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546491    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546495    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546499    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546503    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546508    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546512    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546517    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546521    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546525    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546529    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546534    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:04.558492 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546538    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546542    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546553    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546557    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546561    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546566    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546570    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546574    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546578    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546582    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546586    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546594    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546598    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546603    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546607    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546611    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546615    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546619    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546624    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:04.558986 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546629    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546634    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546638    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546642    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546646    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546650    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546654    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546659    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546663    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:04.546667    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.546758    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.549530    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.550638    2574 server.go:1014] "Starting client certificate rotation"
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.550742    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.550787    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.554897    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:04.559433 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.556463    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:04.569001 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.568987    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:42:04.572249 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.572236    2574 log.go:25] "Validated CRI v1 image API"
Oct 29 22:42:04.573043 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.573017    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:42:04.578135 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.578116    2574 fs.go:135] Filesystem UUIDs: map[3490f51c-f590-4daa-b0c5-67ce535ddea9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b28cd44d-ab8d-47b0-8e93-14356514a3a9:/dev/nvme0n1p4]
Oct 29 22:42:04.578178 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.578136    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 22:42:04.585075 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.584943    2574 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:04.583498307 +0000 UTC m=+0.380265430 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096381 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec4f5cbdf26456910a5b873861821 SystemUUID:ec2ec4f5-cbdf-2645-6910-a5b873861821 BootID:995fd839-fb92-4016-98f3-1d22a369a05f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:fc:96:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:fc:96:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:98:69:5b:02:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:42:04.585796 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.585785    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:42:04.585877 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.585866    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:42:04.589223 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.589198    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:42:04.589352 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.589224    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:42:04.589397 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.589355    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:42:04.589397 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.589363    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:42:04.589448 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.589402    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:04.589448 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.589420    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:04.590195 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.590185    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:42:04.593210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.593198    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:42:04.593326 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.593317    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:42:04.594243 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.594233    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:42:04.594275 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.594248    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:42:04.594275 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.594258    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:42:04.594275 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.594267    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:42:04.594275 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.594275    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:42:04.594607 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.594596    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:04.597814 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.597797    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:42:04.604817 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.604801    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:04.605915 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.605903    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:42:04.605964 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.605934    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:42:04.606087 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606077    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:42:04.606115 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606093    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:42:04.606115 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606099    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:42:04.606115 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606105    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:42:04.606115 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606111    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:42:04.606115 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606116    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:42:04.606796 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606787    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:42:04.606796 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606796    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:42:04.606854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606803    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:42:04.606854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606810    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:42:04.606854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606818    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:42:04.606854 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.606826    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:42:04.607388 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.607380    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:42:04.607424 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.607390    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:42:04.611393 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.611382    2574 server.go:1263] "Started kubelet"
Oct 29 22:42:04.611526 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.611491    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:42:04.611627 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.611481    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:42:04.611627 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.611575    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:42:04.612164 ip-10-0-134-205 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:42:04.613234 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.613218    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:42:04.614689 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.614676    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:42:04.619308 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.619286    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 22:42:04.620922 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.620903    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:42:04.621442 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.621424    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:42:04.621536 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.621518    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:04.622067 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622050    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:42:04.622358 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622340    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:42:04.622358 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622358    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:42:04.622498 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622366    2574 factory.go:55] Registering systemd factory
Oct 29 22:42:04.622498 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622377    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:42:04.622498 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622447    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:42:04.622498 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622456    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:42:04.622839 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622824    2574 factory.go:153] Registering CRI-O factory
Oct 29 22:42:04.622913 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622843    2574 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:42:04.622913 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622895    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:42:04.623006 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622923    2574 factory.go:103] Registering Raw factory
Oct 29 22:42:04.623006 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.622952    2574 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:42:04.624068 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.624055    2574 manager.go:319] Starting recovery of all containers
Oct 29 22:42:04.635143 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.635121    2574 manager.go:324] Recovery completed
Oct 29 22:42:04.641179 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.641016    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:04.642947 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.642927    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:04.643015 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.642960    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:04.643015 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.642973    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:04.643480 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.643462    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:42:04.643480 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.643479    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:42:04.643591 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.643497    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:42:04.643647 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.643480    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:04.644344 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.644325    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:04.644451 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.644359    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:04.644451 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.644371    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:04.654274 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.654257    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:42:04.654366 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.654326    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 22:42:04.654366 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.654330    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-205.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 22:42:04.655601 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.654311    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-205.ec2.internal.18e32154c64ae58f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-205.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-205.ec2.internal,},FirstTimestamp:2026-10-29 22:42:04.611356047 +0000 UTC m=+0.408123172,LastTimestamp:2026-10-29 22:42:04.611356047 +0000 UTC m=+0.408123172,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-205.ec2.internal,}"
Oct 29 22:42:04.669615 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.669544    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-205.ec2.internal.18e32154c82ced80  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-205.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-205.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-205.ec2.internal,},FirstTimestamp:2026-10-29 22:42:04.642946432 +0000 UTC m=+0.439713555,LastTimestamp:2026-10-29 22:42:04.642946432 +0000 UTC m=+0.439713555,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-205.ec2.internal,}"
Oct 29 22:42:04.670329 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.670297    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 22:42:04.670329 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.670297    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 22:42:04.670995 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.670977    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6ckfb"
Oct 29 22:42:04.682721 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.682702    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6ckfb"
Oct 29 22:42:04.703512 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.703377    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-205.ec2.internal.18e32154c82d416d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-205.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-205.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-205.ec2.internal,},FirstTimestamp:2026-10-29 22:42:04.642967917 +0000 UTC m=+0.439735040,LastTimestamp:2026-10-29 22:42:04.642967917 +0000 UTC m=+0.439735040,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-205.ec2.internal,}"
Oct 29 22:42:04.780555 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.703527    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 22:42:04.780555 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.714933    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-205.ec2.internal.18e32154c82d666c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-205.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-205.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-205.ec2.internal,},FirstTimestamp:2026-10-29 22:42:04.642977388 +0000 UTC m=+0.439744511,LastTimestamp:2026-10-29 22:42:04.642977388 +0000 UTC m=+0.439744511,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-205.ec2.internal,}"
Oct 29 22:42:04.780555 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.721978    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:04.780555 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.777126    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-205.ec2.internal.18e32154c82ced80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-205.ec2.internal.18e32154c82ced80  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-205.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-205.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-205.ec2.internal,},FirstTimestamp:2026-10-29 22:42:04.642946432 +0000 UTC m=+0.439713555,LastTimestamp:2026-10-29 22:42:04.644341755 +0000 UTC m=+0.441108889,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-205.ec2.internal,}"
Oct 29 22:42:04.780555 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.779901    2574 policy_none.go:50] "Start"
Oct 29 22:42:04.780555 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.779917    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:42:04.780555 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.779932    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:42:04.782831 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.781145    2574 policy_none.go:44] "Start"
Oct 29 22:42:04.819326 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.819300    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:42:04.822363 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.822342    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:04.823291 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.823279    2574 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:42:04.823386 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.823371    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:42:04.823452 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.823387    2574 server.go:87] "Starting device plugin registration server"
Oct 29 22:42:04.823664 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.823650    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:42:04.823789 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.823666    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:42:04.823789 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.823756    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:42:04.823899 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.823851    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:42:04.823899 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.823858    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:42:04.824536 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.824516    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:42:04.824626 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.824566    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:04.825590 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.825569    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:42:04.826852 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.826835    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:42:04.826934 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.826856    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:42:04.826934 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.826873    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:42:04.826934 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.826905    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 22:42:04.842478 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.842460    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:04.885207 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.885179    2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-205.ec2.internal\" not found" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.923822 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.923800    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:04.924754 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.924736    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:04.924842 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.924782    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:04.924842 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.924797    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:04.924842 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.924821    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.927725 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.927700    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"]
Oct 29 22:42:04.927791 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.927782    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:04.928511 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.928497    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:04.928563 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.928523    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:04.928563 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.928532    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:04.929903 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.929890    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:04.930061 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930033    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.930110 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930063    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:04.930574 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930558    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:04.930574 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930568    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:04.930659 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930586    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:04.930659 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930592    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:04.930659 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930596    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:04.930659 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.930602    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:04.931598 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.931582    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.931677 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.931604    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:04.932262 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.932247    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:04.932343 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.932274    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:04.932343 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.932289    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:04.945253 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:04.945234    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.945253 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.945253    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-205.ec2.internal\": node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:04.961515 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.961488    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-205.ec2.internal\" not found" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.965178 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.965163    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-205.ec2.internal\" not found" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:04.990039 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:04.990019    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.024046 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.024022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.024123 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.024047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.024123 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.024062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cac70c0770f909e7c0598a461e924492-config\") pod \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" (UID: \"cac70c0770f909e7c0598a461e924492\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.090976 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.090906    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.124579 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.124551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.124704 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.124584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.124704 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.124612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cac70c0770f909e7c0598a461e924492-config\") pod \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" (UID: \"cac70c0770f909e7c0598a461e924492\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.124704 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.124644    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.124704 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.124651    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.124704 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.124644    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cac70c0770f909e7c0598a461e924492-config\") pod \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" (UID: \"cac70c0770f909e7c0598a461e924492\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.191750 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.191724    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.264662 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.264640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.269498 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.269482    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:05.292571 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.292451    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.395836 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.395809    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.498329 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.498297    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.551431 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.551399    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 22:42:05.551901 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.551560    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:42:05.599110 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.599049    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.699169 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.699141    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.750786 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.750752    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.752557 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.752533    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.752660 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.752578    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.752660 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.752594    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.780707 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.780669    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:37:04 +0000 UTC" deadline="2028-05-06 16:52:53.070284477 +0000 UTC"
Oct 29 22:42:05.780707 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.780700    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13314h10m47.289588121s"
Oct 29 22:42:05.799918 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.799892    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:05.850181 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:05.850121    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac70c0770f909e7c0598a461e924492.slice/crio-91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e WatchSource:0}: Error finding container 91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e: Status 404 returned error can't find the container with id 91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e
Oct 29 22:42:05.850520 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:05.850498    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b311dfc9e200f9ebd4f84d0618b84c9.slice/crio-f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119 WatchSource:0}: Error finding container f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119: Status 404 returned error can't find the container with id f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119
Oct 29 22:42:05.858958 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:05.858941    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:42:05.900134 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:05.900106    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:06.005999 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.005966    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:06.091187 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.091160    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:06.106237 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.106180    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:06.206828 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.206796    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:06.226881 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.226834    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:06.264457 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.264359    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:06.322218 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.322190    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:06.347685 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.347658    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:06.348993 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.348964    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:06.374413 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.374355    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:06.595672 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.595635    2574 apiserver.go:52] "Watching apiserver"
Oct 29 22:42:06.622302 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.622262    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 22:42:06.629287 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.629225    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:42:06.629587 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.629561    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal","openshift-dns/node-resolver-hg7lt","kube-system/global-pull-secret-syncer-nt7zn","openshift-monitoring/node-exporter-zspc8","openshift-multus/network-metrics-daemon-nfc6s","kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal","openshift-image-registry/node-ca-76p4s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t","openshift-multus/multus-additional-cni-plugins-42xsf","openshift-multus/multus-mcxlq","kube-system/konnectivity-agent-hc2bd","openshift-cluster-node-tuning-operator/tuned-867zt","openshift-network-diagnostics/network-check-target-649bp","openshift-network-operator/iptables-alerter-4vngd","openshift-ovn-kubernetes/ovnkube-node-2sbt6"]
Oct 29 22:42:06.631187 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.631166    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.632624 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.632595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.633892 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.633870    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.633995 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.633956    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:06.635267 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.635247    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.636630 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.636609    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:06.636717 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.636679    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:06.637707 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.637690    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.639226 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.639207    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.640578 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.640555    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.641919 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.641899    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.643388 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.643365    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.644924 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.644742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.646278 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.646254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.647745 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.647726    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:06.647904 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.647791    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:06.649584 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.649553    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6rt5\""
Oct 29 22:42:06.649584 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.649558    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.649707 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.649625    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:42:06.649707 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.649647    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mhfn7\""
Oct 29 22:42:06.649707 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.649695    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.649850 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.649764    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:42:06.650094 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.650077    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:42:06.652860 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.652840    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:42:06.688327 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.688305    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:42:06.688443 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.688363    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l5f5l"
Oct 29 22:42:06.692562 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.692539    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l5f5l"
Oct 29 22:42:06.723973 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.723950    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:42:06.733984 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.733957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.734108 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.733991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-socket-dir-parent\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.734108 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5612c1b5-6517-4a68-a0a4-3a10ff070d31-serviceca\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.734108 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9pw\" (UniqueName: \"kubernetes.io/projected/becd3290-1b8d-45ef-8381-b2aec149f522-kube-api-access-vp9pw\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:06.734279 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzvq\" (UniqueName: \"kubernetes.io/projected/d4b0b131-a728-4cee-aa9e-efa80695c400-kube-api-access-qlzvq\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.734279 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-etc-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.734279 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-node-log\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.734279 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734238    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-bin\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.734500 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734296    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64s69\" (UniqueName: \"kubernetes.io/projected/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-kube-api-access-64s69\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.734500 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734355    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-systemd\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.734500 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5612c1b5-6517-4a68-a0a4-3a10ff070d31-host\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.734500 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-binary-copy\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.734500 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-k8s-cni-cncf-io\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.734757 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-netd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.734757 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734568    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovn-node-metrics-cert\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.734757 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-var-lib-kubelet\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.734757 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734655    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-tls\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.734757 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-etc-kubernetes\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.734757 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrnj\" (UniqueName: \"kubernetes.io/projected/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kube-api-access-9lrnj\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-log-socket\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmrs\" (UniqueName: \"kubernetes.io/projected/5612c1b5-6517-4a68-a0a4-3a10ff070d31-kube-api-access-slmrs\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4b0b131-a728-4cee-aa9e-efa80695c400-hosts-file\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-netns\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.734971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-system-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.735084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735027    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-hostroot\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b0b131-a728-4cee-aa9e-efa80695c400-tmp-dir\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5caa8e9-a021-4107-8fb3-945c7b1179e6-host-slash\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735292    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmnh\" (UniqueName: \"kubernetes.io/projected/be8bd6d3-88a1-482b-a514-d254705c5cbc-kube-api-access-frmnh\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735344    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735396    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-device-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-slash\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-env-overrides\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.735491 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-modprobe-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-cnibin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-etc-tuned\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735655    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-var-lib-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-cni-binary-copy\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-kubelet\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8ea7bf0-7270-447b-8aeb-a91164efe631-agent-certs\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-dbus\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-system-cni-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-cnibin\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-multus\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.735980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-multus-daemon-config\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-config\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.736155 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8ea7bf0-7270-447b-8aeb-a91164efe631-konnectivity-ca\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-wtmp\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736084    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-multus-certs\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vct\" (UniqueName: \"kubernetes.io/projected/b3a24a6b-c505-47ea-9400-f359710c3883-kube-api-access-n5vct\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736160    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7fl\" (UniqueName: \"kubernetes.io/projected/b5caa8e9-a021-4107-8fb3-945c7b1179e6-kube-api-access-xd7fl\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736299    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysconfig\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-kubernetes\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5pbs\" (UniqueName: \"kubernetes.io/projected/c9d9a88e-975d-4596-9c23-1144707d5391-kube-api-access-z5pbs\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-lib-modules\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-root\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pskr\" (UniqueName: \"kubernetes.io/projected/13e0ca11-32d7-4d64-9388-7f50197dd515-kube-api-access-6pskr\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736618    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-bin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-kubelet\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.736983 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736676    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-systemd-units\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-netns\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-conf-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-host\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-kubelet-config\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5caa8e9-a021-4107-8fb3-945c7b1179e6-iptables-alerter-script\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-systemd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-ovn\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.736990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-script-lib\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-conf\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-run\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-sys\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737122    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-tmp\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.737766 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737146    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-sys\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.738577 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-textfile\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.738577 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.737200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-metrics-client-ca\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.759348 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.759322    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.759482 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.759384    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:42:06.759551 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.759526    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:42:06.759618 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.759584    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.759674 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.759629    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.759889 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.759868    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.759992 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.759942    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.760352 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.760333    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.760473 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.760449    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pd9xt\""
Oct 29 22:42:06.760608 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.760593    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.760754 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.760739    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:42:06.760871 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.760858    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:42:06.760968 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.760955    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:42:06.761070 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761048    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.761154 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761077    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:42:06.761218 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761201    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kr4qv\""
Oct 29 22:42:06.761312 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761298    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:42:06.761595 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761579    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:42:06.761682 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761650    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.761745 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761735    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:42:06.761833 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761819    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.761947 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.761933    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.762027 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762014    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9xcm\""
Oct 29 22:42:06.762133 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762119    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:42:06.762200 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762169    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.762259 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762234    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vd45d\""
Oct 29 22:42:06.762452 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762433    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:42:06.762570 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762549    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l2jp\""
Oct 29 22:42:06.762672 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762658    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:42:06.762745 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762707    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:42:06.762806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762785    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wccb2\""
Oct 29 22:42:06.762860 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762837    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rfj6j\""
Oct 29 22:42:06.762911 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762900    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:42:06.762948 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.762920    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:42:06.763039 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.763025    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g9cd8\""
Oct 29 22:42:06.832575 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.832502    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal" event={"ID":"3b311dfc9e200f9ebd4f84d0618b84c9","Type":"ContainerStarted","Data":"f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119"}
Oct 29 22:42:06.833902 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.833875    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal" event={"ID":"cac70c0770f909e7c0598a461e924492","Type":"ContainerStarted","Data":"91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e"}
Oct 29 22:42:06.838028 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.837997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-systemd-units\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838137 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-netns\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.838137 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-conf-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.838137 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838093    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-host\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.838137 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838105    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-systemd-units\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838137 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-kubelet-config\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838167    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-netns\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838220    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-kubelet-config\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838229    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5caa8e9-a021-4107-8fb3-945c7b1179e6-iptables-alerter-script\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838277    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-systemd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-ovn\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838370    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.838399 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838845 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-script-lib\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838845 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838429    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-systemd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838845 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838476    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838845 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838563    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-ovn\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.838845 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838706    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.838845 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-conf\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838863    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-conf-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838872    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-host\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-run\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-sys\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.838971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-tmp\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839023    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-conf\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839019    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5caa8e9-a021-4107-8fb3-945c7b1179e6-iptables-alerter-script\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-sys\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839072    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-sys\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-textfile\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839138    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-sys\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839167    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-run\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-metrics-client-ca\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839229    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-script-lib\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-socket-dir-parent\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5612c1b5-6517-4a68-a0a4-3a10ff070d31-serviceca\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9pw\" (UniqueName: \"kubernetes.io/projected/becd3290-1b8d-45ef-8381-b2aec149f522-kube-api-access-vp9pw\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzvq\" (UniqueName: \"kubernetes.io/projected/d4b0b131-a728-4cee-aa9e-efa80695c400-kube-api-access-qlzvq\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-etc-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839406    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-textfile\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839547    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-node-log\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-bin\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64s69\" (UniqueName: \"kubernetes.io/projected/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-kube-api-access-64s69\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-systemd\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5612c1b5-6517-4a68-a0a4-3a10ff070d31-host\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.839895 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-binary-copy\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-k8s-cni-cncf-io\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-netd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovn-node-metrics-cert\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.839988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-var-lib-kubelet\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-tls\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-etc-kubernetes\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrnj\" (UniqueName: \"kubernetes.io/projected/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kube-api-access-9lrnj\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-log-socket\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmrs\" (UniqueName: \"kubernetes.io/projected/5612c1b5-6517-4a68-a0a4-3a10ff070d31-kube-api-access-slmrs\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840146    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4b0b131-a728-4cee-aa9e-efa80695c400-hosts-file\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.840384 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-netns\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-system-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-hostroot\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5612c1b5-6517-4a68-a0a4-3a10ff070d31-host\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b0b131-a728-4cee-aa9e-efa80695c400-tmp-dir\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840576    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-netns\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5caa8e9-a021-4107-8fb3-945c7b1179e6-host-slash\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840668    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840723    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-metrics-client-ca\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmnh\" (UniqueName: \"kubernetes.io/projected/be8bd6d3-88a1-482b-a514-d254705c5cbc-kube-api-access-frmnh\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.840886 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:06.841528 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-device-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.841528 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.840918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-slash\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.845176 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.845145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-env-overrides\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.845392 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.845375    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-tmp\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.845459 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.845438    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-node-log\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.845521 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.845505    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-var-lib-kubelet\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.845686 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.845662    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-binary-copy\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.846016 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.845993    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-slash\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.846120 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4b0b131-a728-4cee-aa9e-efa80695c400-hosts-file\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.846216 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.846188    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:06.846325 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846223    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-env-overrides\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.846325 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-modprobe-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.846437 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.846347    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret podName:34812142-44fe-4a32-b03e-97fc0a461bd5 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:07.346330611 +0000 UTC m=+3.143097730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret") pod "global-pull-secret-syncer-nt7zn" (UID: "34812142-44fe-4a32-b03e-97fc0a461bd5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:06.846437 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-cnibin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.846437 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-etc-tuned\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.846437 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-var-lib-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846438    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-modprobe-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846469    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846479    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-netd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846509    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-k8s-cni-cncf-io\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-cni-binary-copy\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-kubelet\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.846650 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8ea7bf0-7270-447b-8aeb-a91164efe631-agent-certs\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-dbus\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-system-cni-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846722    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-var-lib-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-cnibin\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846771    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846806    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-cnibin\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846831    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-device-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.847107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.846897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-multus\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.847463 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847321    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.847463 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-multus-daemon-config\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.847463 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847434    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.847463 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-config\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.847681 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8ea7bf0-7270-447b-8aeb-a91164efe631-konnectivity-ca\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.847681 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-wtmp\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.847681 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847558    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-multus-certs\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.847681 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847586    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-multus\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.847910 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847836    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-config\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.847979 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847921    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-cnibin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.848500 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848475    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8ea7bf0-7270-447b-8aeb-a91164efe631-konnectivity-ca\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.848612 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848595    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-wtmp\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.848662 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848646    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-multus-certs\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.848713 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848700    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-systemd\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.848772 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848752    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-socket-dir-parent\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.848828 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848794    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-etc-kubernetes\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.848905 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848858    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.848965 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848930    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-system-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.849021 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.848972    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-hostroot\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.849090 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849068    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-dbus\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:06.849151 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849132    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-bin\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.849205 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849171    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-tls\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.849269 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849252    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b0b131-a728-4cee-aa9e-efa80695c400-tmp-dir\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.849328 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.849266    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:06.849383 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.849331    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs podName:becd3290-1b8d-45ef-8381-b2aec149f522 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:07.349307713 +0000 UTC m=+3.146074846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs") pod "network-metrics-daemon-nfc6s" (UID: "becd3290-1b8d-45ef-8381-b2aec149f522") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:06.849450 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849390    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5caa8e9-a021-4107-8fb3-945c7b1179e6-host-slash\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.849450 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849428    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-kubelet\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.849705 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849664    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-log-socket\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.849782 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849721    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-system-cni-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.849949    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.850018    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.850078    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.850436    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-cni-binary-copy\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.850582    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-etc-tuned\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.850641    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-etc-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.850860    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5612c1b5-6517-4a68-a0a4-3a10ff070d31-serviceca\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.851196    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-multus-daemon-config\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.851806 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.851794    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.847590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vct\" (UniqueName: \"kubernetes.io/projected/b3a24a6b-c505-47ea-9400-f359710c3883-kube-api-access-n5vct\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7fl\" (UniqueName: \"kubernetes.io/projected/b5caa8e9-a021-4107-8fb3-945c7b1179e6-kube-api-access-xd7fl\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysconfig\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852330    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-kubernetes\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pbs\" (UniqueName: \"kubernetes.io/projected/c9d9a88e-975d-4596-9c23-1144707d5391-kube-api-access-z5pbs\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-lib-modules\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-root\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pskr\" (UniqueName: \"kubernetes.io/projected/13e0ca11-32d7-4d64-9388-7f50197dd515-kube-api-access-6pskr\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852483    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovn-node-metrics-cert\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-bin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-kubelet\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852587    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-kubernetes\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852619    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-kubelet\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.854276 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852926    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-lib-modules\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.855052 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.852993    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-root\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.855052 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.853173    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.855052 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.853242    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.855052 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.853447    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysconfig\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.855052 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.853504    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-bin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.855052 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.854938    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8ea7bf0-7270-447b-8aeb-a91164efe631-agent-certs\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.856450 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.856424    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.877997 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.877970    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzvq\" (UniqueName: \"kubernetes.io/projected/d4b0b131-a728-4cee-aa9e-efa80695c400-kube-api-access-qlzvq\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.878890 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.878865    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmrs\" (UniqueName: \"kubernetes.io/projected/5612c1b5-6517-4a68-a0a4-3a10ff070d31-kube-api-access-slmrs\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.881089 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.881007    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrnj\" (UniqueName: \"kubernetes.io/projected/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kube-api-access-9lrnj\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:06.882175 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.881638    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64s69\" (UniqueName: \"kubernetes.io/projected/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-kube-api-access-64s69\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.882175 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.882014    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:06.882175 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.882039    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:06.882175 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.882054    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qvth for pod openshift-network-diagnostics/network-check-target-649bp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:06.882175 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.882113    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vct\" (UniqueName: \"kubernetes.io/projected/b3a24a6b-c505-47ea-9400-f359710c3883-kube-api-access-n5vct\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:06.882175 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:06.882126    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth podName:e0e7869c-ca3a-4a1c-abac-f3bfa1639076 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:07.382107276 +0000 UTC m=+3.178874411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5qvth" (UniqueName: "kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth") pod "network-check-target-649bp" (UID: "e0e7869c-ca3a-4a1c-abac-f3bfa1639076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:06.882921 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.882773    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmnh\" (UniqueName: \"kubernetes.io/projected/be8bd6d3-88a1-482b-a514-d254705c5cbc-kube-api-access-frmnh\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.883134 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.883097    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7fl\" (UniqueName: \"kubernetes.io/projected/b5caa8e9-a021-4107-8fb3-945c7b1179e6-kube-api-access-xd7fl\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:06.884536 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.884508    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9pw\" (UniqueName: \"kubernetes.io/projected/becd3290-1b8d-45ef-8381-b2aec149f522-kube-api-access-vp9pw\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:06.885258 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.885238    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pskr\" (UniqueName: \"kubernetes.io/projected/13e0ca11-32d7-4d64-9388-7f50197dd515-kube-api-access-6pskr\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:06.886199 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.886174    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pbs\" (UniqueName: \"kubernetes.io/projected/c9d9a88e-975d-4596-9c23-1144707d5391-kube-api-access-z5pbs\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:06.914716 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.914692    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:06.945526 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.945497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:06.957187 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.957162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:06.958050 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:06.957938    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5612c1b5_6517_4a68_a0a4_3a10ff070d31.slice/crio-b2b3b8b11d76e465cbb6de9a72c6da60f625ff03a98ae4e40b85823e7fda8939 WatchSource:0}: Error finding container b2b3b8b11d76e465cbb6de9a72c6da60f625ff03a98ae4e40b85823e7fda8939: Status 404 returned error can't find the container with id b2b3b8b11d76e465cbb6de9a72c6da60f625ff03a98ae4e40b85823e7fda8939
Oct 29 22:42:06.966629 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:06.966598    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ea7bf0_7270_447b_8aeb_a91164efe631.slice/crio-d6e3908315ccab466925439ffe555999a3b7490ae2c51847dea467ab87315baa WatchSource:0}: Error finding container d6e3908315ccab466925439ffe555999a3b7490ae2c51847dea467ab87315baa: Status 404 returned error can't find the container with id d6e3908315ccab466925439ffe555999a3b7490ae2c51847dea467ab87315baa
Oct 29 22:42:06.969374 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.969349    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:06.978780 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:06.978754    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8bd6d3_88a1_482b_a514_d254705c5cbc.slice/crio-378dfaabbdb187afb5199200e0dd36d222d53a287e3723a79b3f838f25e767ba WatchSource:0}: Error finding container 378dfaabbdb187afb5199200e0dd36d222d53a287e3723a79b3f838f25e767ba: Status 404 returned error can't find the container with id 378dfaabbdb187afb5199200e0dd36d222d53a287e3723a79b3f838f25e767ba
Oct 29 22:42:06.982679 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.982659    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:06.983956 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.983935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:06.991981 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:06.991956    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8ed141_dd58_45a2_841e_22c1d8eb19c1.slice/crio-796ee053bd9e33bfbe6c068f283ddbe307a4af4a1083a198c432116098756199 WatchSource:0}: Error finding container 796ee053bd9e33bfbe6c068f283ddbe307a4af4a1083a198c432116098756199: Status 404 returned error can't find the container with id 796ee053bd9e33bfbe6c068f283ddbe307a4af4a1083a198c432116098756199
Oct 29 22:42:06.996861 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:06.996821    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b0b131_a728_4cee_aa9e_efa80695c400.slice/crio-0da6962a6847d0f181a2713c35b65d61958ba76ec442c4577b279434dc3344b9 WatchSource:0}: Error finding container 0da6962a6847d0f181a2713c35b65d61958ba76ec442c4577b279434dc3344b9: Status 404 returned error can't find the container with id 0da6962a6847d0f181a2713c35b65d61958ba76ec442c4577b279434dc3344b9
Oct 29 22:42:06.997737 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.997717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:07.000065 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:06.999797    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:07.007293 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.007272    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:07.007460 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:07.007435    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e0ca11_32d7_4d64_9388_7f50197dd515.slice/crio-8da461694f00fde0c4bb4751c27a73b36796e93809e583c1d566923daad2f04b WatchSource:0}: Error finding container 8da461694f00fde0c4bb4751c27a73b36796e93809e583c1d566923daad2f04b: Status 404 returned error can't find the container with id 8da461694f00fde0c4bb4751c27a73b36796e93809e583c1d566923daad2f04b
Oct 29 22:42:07.010926 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:07.010874    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5caa8e9_a021_4107_8fb3_945c7b1179e6.slice/crio-c8d946a20c0900abd0a1d268bc63105e0f1c69ed7f9c1285da30aa1bfe2b8503 WatchSource:0}: Error finding container c8d946a20c0900abd0a1d268bc63105e0f1c69ed7f9c1285da30aa1bfe2b8503: Status 404 returned error can't find the container with id c8d946a20c0900abd0a1d268bc63105e0f1c69ed7f9c1285da30aa1bfe2b8503
Oct 29 22:42:07.017012 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:07.016986    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a24a6b_c505_47ea_9400_f359710c3883.slice/crio-0b196787d199bfdb333da3fb04debaef4400bb6cde6b235f38c117b78b1c2630 WatchSource:0}: Error finding container 0b196787d199bfdb333da3fb04debaef4400bb6cde6b235f38c117b78b1c2630: Status 404 returned error can't find the container with id 0b196787d199bfdb333da3fb04debaef4400bb6cde6b235f38c117b78b1c2630
Oct 29 22:42:07.018192 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.018054    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:07.022976 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.022955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:07.027924 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:07.027897    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1014721_61ff_4d6e_b0f7_83b4f51649c5.slice/crio-79d3f2b99ef3f097f6215c4b1af730b6990fdf6a7bb744610529a34c9ef05e41 WatchSource:0}: Error finding container 79d3f2b99ef3f097f6215c4b1af730b6990fdf6a7bb744610529a34c9ef05e41: Status 404 returned error can't find the container with id 79d3f2b99ef3f097f6215c4b1af730b6990fdf6a7bb744610529a34c9ef05e41
Oct 29 22:42:07.033158 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:07.033134    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d9a88e_975d_4596_9c23_1144707d5391.slice/crio-de49e001a30f2239a57f2f5bb5b3754ec9b0d7180efa75cb69a7872d73264603 WatchSource:0}: Error finding container de49e001a30f2239a57f2f5bb5b3754ec9b0d7180efa75cb69a7872d73264603: Status 404 returned error can't find the container with id de49e001a30f2239a57f2f5bb5b3754ec9b0d7180efa75cb69a7872d73264603
Oct 29 22:42:07.357366 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.357333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:07.357490 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.357442    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:07.357569 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.357555    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:07.357631 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.357619    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret podName:34812142-44fe-4a32-b03e-97fc0a461bd5 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:08.357601181 +0000 UTC m=+4.154368309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret") pod "global-pull-secret-syncer-nt7zn" (UID: "34812142-44fe-4a32-b03e-97fc0a461bd5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:07.357964 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.357946    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:07.358063 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.357996    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs podName:becd3290-1b8d-45ef-8381-b2aec149f522 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:08.357981945 +0000 UTC m=+4.154749077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs") pod "network-metrics-daemon-nfc6s" (UID: "becd3290-1b8d-45ef-8381-b2aec149f522") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:07.459301 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.458423    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:07.459301 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.458533    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:07.459301 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.458545    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:07.459301 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.458554    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qvth for pod openshift-network-diagnostics/network-check-target-649bp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:07.459301 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.458590    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth podName:e0e7869c-ca3a-4a1c-abac-f3bfa1639076 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:08.458578023 +0000 UTC m=+4.255345147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qvth" (UniqueName: "kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth") pod "network-check-target-649bp" (UID: "e0e7869c-ca3a-4a1c-abac-f3bfa1639076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:07.695641 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.695302    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:06 +0000 UTC" deadline="2028-07-31 05:19:06.871592284 +0000 UTC"
Oct 29 22:42:07.695641 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.695568    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15366h36m59.176028957s"
Oct 29 22:42:07.827898 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.827785    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:07.828058 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.827919    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:07.828345 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.828325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:07.828438 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:07.828420    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:07.848896 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.848861    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-867zt" event={"ID":"c9d9a88e-975d-4596-9c23-1144707d5391","Type":"ContainerStarted","Data":"de49e001a30f2239a57f2f5bb5b3754ec9b0d7180efa75cb69a7872d73264603"}
Oct 29 22:42:07.856166 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.856125    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vngd" event={"ID":"b5caa8e9-a021-4107-8fb3-945c7b1179e6","Type":"ContainerStarted","Data":"c8d946a20c0900abd0a1d268bc63105e0f1c69ed7f9c1285da30aa1bfe2b8503"}
Oct 29 22:42:07.875137 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.875104    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerStarted","Data":"8da461694f00fde0c4bb4751c27a73b36796e93809e583c1d566923daad2f04b"}
Oct 29 22:42:07.877738 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.877706    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg7lt" event={"ID":"d4b0b131-a728-4cee-aa9e-efa80695c400","Type":"ContainerStarted","Data":"0da6962a6847d0f181a2713c35b65d61958ba76ec442c4577b279434dc3344b9"}
Oct 29 22:42:07.895018 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.894960    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zspc8" event={"ID":"be8bd6d3-88a1-482b-a514-d254705c5cbc","Type":"ContainerStarted","Data":"378dfaabbdb187afb5199200e0dd36d222d53a287e3723a79b3f838f25e767ba"}
Oct 29 22:42:07.907613 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.907578    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hc2bd" event={"ID":"c8ea7bf0-7270-447b-8aeb-a91164efe631","Type":"ContainerStarted","Data":"d6e3908315ccab466925439ffe555999a3b7490ae2c51847dea467ab87315baa"}
Oct 29 22:42:07.928193 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.928158    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76p4s" event={"ID":"5612c1b5-6517-4a68-a0a4-3a10ff070d31","Type":"ContainerStarted","Data":"b2b3b8b11d76e465cbb6de9a72c6da60f625ff03a98ae4e40b85823e7fda8939"}
Oct 29 22:42:07.937707 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.937675    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal" event={"ID":"cac70c0770f909e7c0598a461e924492","Type":"ContainerStarted","Data":"db8a6451d122c0ca64124b40dad5ca270910cded34e85c9bf425b700202d68fa"}
Oct 29 22:42:07.947438 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.947407    2574 generic.go:356] "Generic (PLEG): container finished" podID="3b311dfc9e200f9ebd4f84d0618b84c9" containerID="604ee44d6201d386ab4ca8fa62615ed3b7235713754da1f2857eb55fbc40e09c" exitCode=0
Oct 29 22:42:07.947547 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.947505    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal" event={"ID":"3b311dfc9e200f9ebd4f84d0618b84c9","Type":"ContainerDied","Data":"604ee44d6201d386ab4ca8fa62615ed3b7235713754da1f2857eb55fbc40e09c"}
Oct 29 22:42:07.957925 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.957862    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t" event={"ID":"d1014721-61ff-4d6e-b0f7-83b4f51649c5","Type":"ContainerStarted","Data":"79d3f2b99ef3f097f6215c4b1af730b6990fdf6a7bb744610529a34c9ef05e41"}
Oct 29 22:42:07.969330 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.969308    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mcxlq" event={"ID":"b3a24a6b-c505-47ea-9400-f359710c3883","Type":"ContainerStarted","Data":"0b196787d199bfdb333da3fb04debaef4400bb6cde6b235f38c117b78b1c2630"}
Oct 29 22:42:07.973828 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.973577    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"796ee053bd9e33bfbe6c068f283ddbe307a4af4a1083a198c432116098756199"}
Oct 29 22:42:07.986506 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:07.985512    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal" podStartSLOduration=1.9854979940000002 podStartE2EDuration="1.985497994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:07.985351909 +0000 UTC m=+3.782119035" watchObservedRunningTime="2026-10-29 22:42:07.985497994 +0000 UTC m=+3.782265130"
Oct 29 22:42:08.368858 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:08.368025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:08.368858 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:08.368081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:08.368858 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.368259    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:08.368858 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.368327    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs podName:becd3290-1b8d-45ef-8381-b2aec149f522 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:10.368307493 +0000 UTC m=+6.165074626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs") pod "network-metrics-daemon-nfc6s" (UID: "becd3290-1b8d-45ef-8381-b2aec149f522") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:08.368858 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.368704    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:08.368858 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.368754    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret podName:34812142-44fe-4a32-b03e-97fc0a461bd5 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:10.36873828 +0000 UTC m=+6.165505414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret") pod "global-pull-secret-syncer-nt7zn" (UID: "34812142-44fe-4a32-b03e-97fc0a461bd5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:08.468876 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:08.468838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:08.469072 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.469056    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:08.469133 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.469076    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:08.469133 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.469089    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qvth for pod openshift-network-diagnostics/network-check-target-649bp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:08.469233 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.469136    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth podName:e0e7869c-ca3a-4a1c-abac-f3bfa1639076 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:10.469122966 +0000 UTC m=+6.265890078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qvth" (UniqueName: "kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth") pod "network-check-target-649bp" (UID: "e0e7869c-ca3a-4a1c-abac-f3bfa1639076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:08.696564 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:08.696481    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:06 +0000 UTC" deadline="2028-05-01 15:02:32.900837324 +0000 UTC"
Oct 29 22:42:08.696564 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:08.696520    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13192h20m24.2043232s"
Oct 29 22:42:08.828791 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:08.827897    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:08.828791 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:08.828030    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:09.000099 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:09.000019    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal" event={"ID":"3b311dfc9e200f9ebd4f84d0618b84c9","Type":"ContainerStarted","Data":"59c78118a8e9da706415c2a6ac40715eddd268444a264357ff224ff1992f2042"}
Oct 29 22:42:09.828026 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:09.828002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:09.828397 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:09.828119    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:09.828465 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:09.828449    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:09.828586 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:09.828533    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:10.004205 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.004161    2574 generic.go:356] "Generic (PLEG): container finished" podID="be8bd6d3-88a1-482b-a514-d254705c5cbc" containerID="ce4853e825378dbc905f1991e44b9be4c4bce003416234177b199fa8ed67f9b8" exitCode=0
Oct 29 22:42:10.004381 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.004275    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zspc8" event={"ID":"be8bd6d3-88a1-482b-a514-d254705c5cbc","Type":"ContainerDied","Data":"ce4853e825378dbc905f1991e44b9be4c4bce003416234177b199fa8ed67f9b8"}
Oct 29 22:42:10.023972 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.023941    2574 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="8999d5772e6d2e6dc893ddaa2f8d4236c1d293ab726a742ccb9f0e887ef8ff03" exitCode=0
Oct 29 22:42:10.024988 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.024959    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerDied","Data":"8999d5772e6d2e6dc893ddaa2f8d4236c1d293ab726a742ccb9f0e887ef8ff03"}
Oct 29 22:42:10.052225 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.052167    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal" podStartSLOduration=4.05214877 podStartE2EDuration="4.05214877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:09.049746159 +0000 UTC m=+4.846513295" watchObservedRunningTime="2026-10-29 22:42:10.05214877 +0000 UTC m=+5.848915907"
Oct 29 22:42:10.220113 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.220071    2574 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8bd6d3_88a1_482b_a514_d254705c5cbc.slice/crio-0dc33beef4fd83e07120e77066bba2f2e0c89714cb73b44cb2ce291c2a2d901d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:42:10.386502 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.385737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:10.386502 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.385803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:10.386502 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.385954    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:10.386502 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.386013    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs podName:becd3290-1b8d-45ef-8381-b2aec149f522 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.385996025 +0000 UTC m=+10.182763141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs") pod "network-metrics-daemon-nfc6s" (UID: "becd3290-1b8d-45ef-8381-b2aec149f522") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:10.386502 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.386383    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:10.386502 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.386427    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret podName:34812142-44fe-4a32-b03e-97fc0a461bd5 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.386413021 +0000 UTC m=+10.183180151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret") pod "global-pull-secret-syncer-nt7zn" (UID: "34812142-44fe-4a32-b03e-97fc0a461bd5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:10.486891 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.486755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:10.487062 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.486940    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:10.487062 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.486959    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:10.487062 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.486972    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qvth for pod openshift-network-diagnostics/network-check-target-649bp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:10.487062 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.487032    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth podName:e0e7869c-ca3a-4a1c-abac-f3bfa1639076 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.487014021 +0000 UTC m=+10.283781155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qvth" (UniqueName: "kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth") pod "network-check-target-649bp" (UID: "e0e7869c-ca3a-4a1c-abac-f3bfa1639076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:10.829476 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:10.829407    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:10.829830 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:10.829527    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:11.037370 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:11.036485    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zspc8" event={"ID":"be8bd6d3-88a1-482b-a514-d254705c5cbc","Type":"ContainerStarted","Data":"0dc33beef4fd83e07120e77066bba2f2e0c89714cb73b44cb2ce291c2a2d901d"}
Oct 29 22:42:11.037370 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:11.036548    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zspc8" event={"ID":"be8bd6d3-88a1-482b-a514-d254705c5cbc","Type":"ContainerStarted","Data":"c6d863bdfe70ee243f853e5c2a1b184490b6f01f722e11d5cd0218c66b482fac"}
Oct 29 22:42:11.084134 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:11.084038    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zspc8" podStartSLOduration=3.788314201 podStartE2EDuration="6.084023646s" totalImagesPullingTime="2.21145504s" totalInitContainerRuntime="84.254405ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:11.083790464 +0000 UTC m=+6.880557805" watchObservedRunningTime="2026-10-29 22:42:11.084023646 +0000 UTC m=+6.880790779"
Oct 29 22:42:11.828092 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:11.827462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:11.828092 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:11.827605    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:11.828092 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:11.827665    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:11.828092 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:11.827708    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:12.050893 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:12.050861    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hc2bd" event={"ID":"c8ea7bf0-7270-447b-8aeb-a91164efe631","Type":"ContainerStarted","Data":"c805d7f11d89cfc8aeec87d35367ee0dcf591a862eb8cf9b96baa3f5375afb16"}
Oct 29 22:42:12.094025 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:12.093345    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hc2bd" podStartSLOduration=2.931924141 podStartE2EDuration="7.093324945s" totalImagesPullingTime="4.161400804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:12.092907461 +0000 UTC m=+7.889674610" watchObservedRunningTime="2026-10-29 22:42:12.093324945 +0000 UTC m=+7.890092080"
Oct 29 22:42:12.829921 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:12.827819    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:12.829921 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:12.827974    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:13.005713 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:13.005510    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:13.006429 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:13.006127    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:13.056076 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:13.055927    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76p4s" event={"ID":"5612c1b5-6517-4a68-a0a4-3a10ff070d31","Type":"ContainerStarted","Data":"4dee1471965cc71e13a43a3a6ed9c965f6ca08541167af22adfd61edbeaf27d0"}
Oct 29 22:42:13.062851 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:13.062824    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t" event={"ID":"d1014721-61ff-4d6e-b0f7-83b4f51649c5","Type":"ContainerStarted","Data":"52551ed6137c6058b3823571461df8894c04f7b4d5446249a7fcef4f88e062bc"}
Oct 29 22:42:13.827721 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:13.827056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:13.827721 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:13.827196    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:13.827721 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:13.827583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:13.827721 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:13.827663    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:14.065924 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.065487    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:14.379162 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.379045    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:42:14.422697 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.422661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:14.422849 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.422712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:14.422849 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.422845    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:14.422916 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.422885    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs podName:becd3290-1b8d-45ef-8381-b2aec149f522 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.422873123 +0000 UTC m=+18.219640234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs") pod "network-metrics-daemon-nfc6s" (UID: "becd3290-1b8d-45ef-8381-b2aec149f522") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:14.423225 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.423130    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:14.423225 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.423161    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret podName:34812142-44fe-4a32-b03e-97fc0a461bd5 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.423152203 +0000 UTC m=+18.219919314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret") pod "global-pull-secret-syncer-nt7zn" (UID: "34812142-44fe-4a32-b03e-97fc0a461bd5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:14.523107 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.523027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:14.523252 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.523215    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:14.523252 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.523230    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:14.523252 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.523243    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qvth for pod openshift-network-diagnostics/network-check-target-649bp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:14.523363 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.523292    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth podName:e0e7869c-ca3a-4a1c-abac-f3bfa1639076 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.523276651 +0000 UTC m=+18.320043777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qvth" (UniqueName: "kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth") pod "network-check-target-649bp" (UID: "e0e7869c-ca3a-4a1c-abac-f3bfa1639076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:14.829550 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.828068    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:14.379064335Z","UUID":"d226c88b-0b02-4f2e-89e5-5b2ef4dd415d","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:42:14.829550 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.828907    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:14.829550 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.829042    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:14.838506 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.838484    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:42:14.838600 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:14.838517    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:42:14.839269 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:14.839203    2574 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e0ca11_32d7_4d64_9388_7f50197dd515.slice/crio-conmon-b9256bb962691ed927dd09792bbc62e2840cf2e8132fa2451be1b56ae6512bf8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e0ca11_32d7_4d64_9388_7f50197dd515.slice/crio-b9256bb962691ed927dd09792bbc62e2840cf2e8132fa2451be1b56ae6512bf8.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:42:15.072520 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:15.072485    2574 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="b9256bb962691ed927dd09792bbc62e2840cf2e8132fa2451be1b56ae6512bf8" exitCode=0
Oct 29 22:42:15.073201 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:15.072596    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerDied","Data":"b9256bb962691ed927dd09792bbc62e2840cf2e8132fa2451be1b56ae6512bf8"}
Oct 29 22:42:15.077893 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:15.077865    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t" event={"ID":"d1014721-61ff-4d6e-b0f7-83b4f51649c5","Type":"ContainerStarted","Data":"6095b96a84e493c5bb982c5cfac0fc922841f5f48ba58adf193236f4b286494c"}
Oct 29 22:42:15.124822 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:15.124750    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-76p4s" podStartSLOduration=5.036372676 podStartE2EDuration="10.124729723s" totalImagesPullingTime="5.088357047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:13.098904972 +0000 UTC m=+8.895672127" watchObservedRunningTime="2026-10-29 22:42:15.124729723 +0000 UTC m=+10.921496857"
Oct 29 22:42:15.827576 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:15.827547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:15.827690 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:15.827593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:15.827690 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:15.827671    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:15.827824 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:15.827780    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:16.083574 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.083533    2574 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="1107f18e808c1716b47484cef81eb5b112f4ea63f79578b484b59a14daa32694" exitCode=0
Oct 29 22:42:16.084131 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.083627    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerDied","Data":"1107f18e808c1716b47484cef81eb5b112f4ea63f79578b484b59a14daa32694"}
Oct 29 22:42:16.086547 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.086520    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg7lt" event={"ID":"d4b0b131-a728-4cee-aa9e-efa80695c400","Type":"ContainerStarted","Data":"e2233b3915b0a437a40dcc112c3a9cd050bb3fc60604cd6749e2af8f1c669429"}
Oct 29 22:42:16.088863 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.088841    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t" event={"ID":"d1014721-61ff-4d6e-b0f7-83b4f51649c5","Type":"ContainerStarted","Data":"f921c6fc95737bb0ba9bb3e439e4a48efddcf1da98e04d37284a4058ce766a61"}
Oct 29 22:42:16.090488 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.090462    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mcxlq" event={"ID":"b3a24a6b-c505-47ea-9400-f359710c3883","Type":"ContainerStarted","Data":"f6543146f714350b03adcf7aa8daf3820c61c7726ea98a7ae41ba411ea7cb9d3"}
Oct 29 22:42:16.220285 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.220234    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hg7lt" podStartSLOduration=3.24842832 podStartE2EDuration="11.220215633s" totalImagesPullingTime="7.971787313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.175120495 +0000 UTC m=+11.971887630" watchObservedRunningTime="2026-10-29 22:42:16.220215633 +0000 UTC m=+12.016982766"
Oct 29 22:42:16.221084 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.221042    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t" podStartSLOduration=3.181454997 podStartE2EDuration="11.221014061s" totalImagesPullingTime="8.039559064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.219835566 +0000 UTC m=+12.016602698" watchObservedRunningTime="2026-10-29 22:42:16.221014061 +0000 UTC m=+12.017781194"
Oct 29 22:42:16.283753 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.283700    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mcxlq" podStartSLOduration=3.293637303 podStartE2EDuration="11.283685537s" totalImagesPullingTime="7.990048234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.283683 +0000 UTC m=+12.080450134" watchObservedRunningTime="2026-10-29 22:42:16.283685537 +0000 UTC m=+12.080452671"
Oct 29 22:42:16.828224 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:16.828190    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:16.828379 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:16.828336    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:17.094330 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:17.094253    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vngd" event={"ID":"b5caa8e9-a021-4107-8fb3-945c7b1179e6","Type":"ContainerStarted","Data":"674223ff427a5257f8d9708140804af2b5fb2c5e143b15c5aee1fb7842eed628"}
Oct 29 22:42:17.097090 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:17.097065    2574 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="34ba4b1a193542cef525f3d64b95c91296ac60366b6c78659a4150e7c829457e" exitCode=0
Oct 29 22:42:17.097207 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:17.097157    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerDied","Data":"34ba4b1a193542cef525f3d64b95c91296ac60366b6c78659a4150e7c829457e"}
Oct 29 22:42:17.136660 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:17.136613    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4vngd" podStartSLOduration=4.348242572 podStartE2EDuration="12.136599686s" totalImagesPullingTime="7.788357114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:17.136262759 +0000 UTC m=+12.933029891" watchObservedRunningTime="2026-10-29 22:42:17.136599686 +0000 UTC m=+12.933366818"
Oct 29 22:42:17.827892 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:17.827591    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:17.828042 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:17.827640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:17.828042 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:17.827966    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:17.828042 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:17.828027    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:18.101793 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:18.101694    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-867zt" event={"ID":"c9d9a88e-975d-4596-9c23-1144707d5391","Type":"ContainerStarted","Data":"f0ce312d10b673477655059e776fe3f768eec6cfe4857c7c9d492380fd4017ec"}
Oct 29 22:42:18.827992 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:18.827954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:18.828172 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:18.828066    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:19.827867 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:19.827834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:19.828342 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:19.827879    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:19.828342 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:19.827961    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:19.828342 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:19.828021    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:20.109715 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:20.109643    2574 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="820eb9f4bb81f331ec3d0c4142d2370bf808f10c346c84ae9b61ed49af69e238" exitCode=0
Oct 29 22:42:20.109715 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:20.109695    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerDied","Data":"820eb9f4bb81f331ec3d0c4142d2370bf808f10c346c84ae9b61ed49af69e238"}
Oct 29 22:42:20.187924 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:20.187881    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-867zt" podStartSLOduration=4.944543706 podStartE2EDuration="15.187867984s" totalImagesPullingTime="10.243324278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:18.146752809 +0000 UTC m=+13.943519942" watchObservedRunningTime="2026-10-29 22:42:20.187867984 +0000 UTC m=+15.984635117"
Oct 29 22:42:20.827304 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:20.827275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:20.827484 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:20.827400    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:21.114537 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:21.114468    2574 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="f7a753d2e725376cf1090115b0a9f14ddc21f02325b9b3a98a0def990e7ac8b4" exitCode=0
Oct 29 22:42:21.114537 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:21.114520    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerDied","Data":"f7a753d2e725376cf1090115b0a9f14ddc21f02325b9b3a98a0def990e7ac8b4"}
Oct 29 22:42:21.828060 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:21.827876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:21.828208 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:21.827928    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:21.828208 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:21.828129    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:21.828208 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:21.828188    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:22.122443 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.122403    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42xsf" event={"ID":"13e0ca11-32d7-4d64-9388-7f50197dd515","Type":"ContainerStarted","Data":"f53990abb12bd2a5810440c1141d1a00c87065b7e4073ab432798e1313dcb1f6"}
Oct 29 22:42:22.125171 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125152    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:42:22.125462 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125444    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc8ed141-dd58-45a2-841e-22c1d8eb19c1" containerID="c1176da34524362fe16ee02ba33776ca5fc4c74d374a424798859d84698981c0" exitCode=1
Oct 29 22:42:22.125519 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125476    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"3cd750bff8d54532ead7182c1f71d8b8e65de3bc389b885c4e1afa9263d7149a"}
Oct 29 22:42:22.125567 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125556    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"39c00254d38357ce654e1e408e80af386cbc31be83998171df2a8a0f08616b20"}
Oct 29 22:42:22.125603 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125588    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"874050cc929f71245393fbe8f9fb4d6442fc740a336e64cb8b92c799339fcd2d"}
Oct 29 22:42:22.125630 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125617    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"c778991ee0e79919cf813ce19cedc69e2a364f962690c17798f32e01a79cc9ab"}
Oct 29 22:42:22.125658 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125644    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerDied","Data":"c1176da34524362fe16ee02ba33776ca5fc4c74d374a424798859d84698981c0"}
Oct 29 22:42:22.125685 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.125671    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"37da88cdc7649c1d512231eae242a83c297f51b2a641b5c2d410bd4d95c917c1"}
Oct 29 22:42:22.174747 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.174705    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-42xsf" podStartSLOduration=7.037571559 podStartE2EDuration="17.174693605s" totalImagesPullingTime="10.137122046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:22.174530674 +0000 UTC m=+17.971297806" watchObservedRunningTime="2026-10-29 22:42:22.174693605 +0000 UTC m=+17.971460738"
Oct 29 22:42:22.485593 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.485519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:22.485593 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.485554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:22.485761 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.485648    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:22.485761 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.485689    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs podName:becd3290-1b8d-45ef-8381-b2aec149f522 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:38.485676798 +0000 UTC m=+34.282443909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs") pod "network-metrics-daemon-nfc6s" (UID: "becd3290-1b8d-45ef-8381-b2aec149f522") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:22.485761 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.485648    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:22.485761 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.485757    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret podName:34812142-44fe-4a32-b03e-97fc0a461bd5 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:38.485746256 +0000 UTC m=+34.282513370 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret") pod "global-pull-secret-syncer-nt7zn" (UID: "34812142-44fe-4a32-b03e-97fc0a461bd5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:22.585820 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.585793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:22.585937 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.585913    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:22.585937 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.585931    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:22.585992 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.585962    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qvth for pod openshift-network-diagnostics/network-check-target-649bp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:22.586036 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.586004    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth podName:e0e7869c-ca3a-4a1c-abac-f3bfa1639076 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:38.585991854 +0000 UTC m=+34.382758965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qvth" (UniqueName: "kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth") pod "network-check-target-649bp" (UID: "e0e7869c-ca3a-4a1c-abac-f3bfa1639076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:22.827460 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:22.827388    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:22.827599 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:22.827511    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:23.827855 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:23.827826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:23.827855 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:23.827854    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:23.828246 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:23.827935    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:23.828246 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:23.828002    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:24.132947 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:24.132876    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:42:24.133219 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:24.133190    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"694ec162a2f1d9b4d7ff9f3410036ee65009155413c3476ab30583828736937d"}
Oct 29 22:42:24.830210 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:24.830184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:24.830593 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:24.830277    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:25.827435 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:25.827416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:25.827529 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:25.827517    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:25.827592 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:25.827578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:25.828075 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:25.828037    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:26.142226 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:26.141971    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:42:26.142687 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:26.142484    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"e121bbe22c73b4499e5ac648f0ad530b579c25c0971f6f439398b05d88c2f03f"}
Oct 29 22:42:26.142807 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:26.142787    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:26.142964 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:26.142950    2574 scope.go:122] "RemoveContainer" containerID="c1176da34524362fe16ee02ba33776ca5fc4c74d374a424798859d84698981c0"
Oct 29 22:42:26.158720 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:26.158702    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:26.827841 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:26.827809    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:26.828007 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:26.827922    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:27.148282 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.148219    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:42:27.148688 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.148615    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" event={"ID":"dc8ed141-dd58-45a2-841e-22c1d8eb19c1","Type":"ContainerStarted","Data":"45eef77961bcc3809cdb1d808ac47a78cd2062e493520e1b8103d13ef9f108a9"}
Oct 29 22:42:27.148900 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.148886    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:27.148977 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.148960    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:27.163729 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.163711    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:27.246609 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.246562    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6" podStartSLOduration=7.967524139 podStartE2EDuration="22.246546661s" totalImagesPullingTime="14.279022522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:27.245753016 +0000 UTC m=+23.042520148" watchObservedRunningTime="2026-10-29 22:42:27.246546661 +0000 UTC m=+23.043313797"
Oct 29 22:42:27.828169 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.828131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:27.828315 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.828173    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:27.828315 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:27.828250    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:27.828413 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:27.828395    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:27.944541 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.944190    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nt7zn"]
Oct 29 22:42:27.945338 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.945306    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-649bp"]
Oct 29 22:42:27.946029 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.946009    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nfc6s"]
Oct 29 22:42:27.946126 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:27.946115    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:27.946249 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:27.946222    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:28.150737 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:28.150706    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:28.151237 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:28.150718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:28.151237 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:28.150818    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:28.151237 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:28.150939    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:29.827713 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:29.827684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:29.827713 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:29.827723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:29.828242 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:29.827824    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:29.828242 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:29.827840    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:29.828242 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:29.827922    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:29.828242 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:29.827989    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:31.251605 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:31.251579    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:31.252111 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:31.251719    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:31.252179 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:31.252163    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:31.827691 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:31.827659    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:31.827691 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:31.827688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:31.827979 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:31.827796    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:31.827979 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:31.827791    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt7zn" podUID="34812142-44fe-4a32-b03e-97fc0a461bd5"
Oct 29 22:42:31.827979 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:31.827883    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfc6s" podUID="becd3290-1b8d-45ef-8381-b2aec149f522"
Oct 29 22:42:31.827979 ip-10-0-134-205 kubenswrapper[2574]: E1029 22:42:31.827959    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-649bp" podUID="e0e7869c-ca3a-4a1c-abac-f3bfa1639076"
Oct 29 22:42:33.447703 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.447511    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeReady"
Oct 29 22:42:33.448033 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.447788    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 22:42:33.509692 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.509667    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f67rh"]
Oct 29 22:42:33.512740 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.512719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.513087 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.513068    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7xwpt"]
Oct 29 22:42:33.515891 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.515877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.534654 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.534632    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f67rh"]
Oct 29 22:42:33.537254 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.537234    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:42:33.537690 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.537672    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:42:33.547493 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.547469    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:42:33.547583 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.547565    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:42:33.547632 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.547579    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzwzj\""
Oct 29 22:42:33.547719 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.547707    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c4xfj\""
Oct 29 22:42:33.548297 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.548283    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vqfnh"]
Oct 29 22:42:33.550284 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.550267    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:42:33.550371 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.550360    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:42:33.551038 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.551024    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.551372 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.551351    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7xwpt"]
Oct 29 22:42:33.566682 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.566662    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqfnh"]
Oct 29 22:42:33.637235 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.637203    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:42:33.637408 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.637397    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:42:33.637502 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.637466    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg4pb\""
Oct 29 22:42:33.637546 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.637509    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:42:33.675342 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79884975-9bf8-48b4-89c1-95e4f33e8285-tmp-dir\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.675342 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/368c881a-7e15-4fad-8acf-2253c547cc54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.675527 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/368c881a-7e15-4fad-8acf-2253c547cc54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.675527 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fm4\" (UniqueName: \"kubernetes.io/projected/368c881a-7e15-4fad-8acf-2253c547cc54-kube-api-access-b4fm4\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.675527 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5tr\" (UniqueName: \"kubernetes.io/projected/79884975-9bf8-48b4-89c1-95e4f33e8285-kube-api-access-6c5tr\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.675527 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/368c881a-7e15-4fad-8acf-2253c547cc54-crio-socket\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.675527 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675524    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79884975-9bf8-48b4-89c1-95e4f33e8285-config-volume\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.675691 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/368c881a-7e15-4fad-8acf-2253c547cc54-data-volume\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.675691 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79884975-9bf8-48b4-89c1-95e4f33e8285-metrics-tls\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.675691 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxw7\" (UniqueName: \"kubernetes.io/projected/8d5029cc-b18a-4547-9e9e-7a172b56e789-kube-api-access-xdxw7\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.675691 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.675670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d5029cc-b18a-4547-9e9e-7a172b56e789-cert\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.776231 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d5029cc-b18a-4547-9e9e-7a172b56e789-cert\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.776231 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79884975-9bf8-48b4-89c1-95e4f33e8285-tmp-dir\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.776346 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/368c881a-7e15-4fad-8acf-2253c547cc54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.776346 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/368c881a-7e15-4fad-8acf-2253c547cc54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.776346 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fm4\" (UniqueName: \"kubernetes.io/projected/368c881a-7e15-4fad-8acf-2253c547cc54-kube-api-access-b4fm4\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.776346 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5tr\" (UniqueName: \"kubernetes.io/projected/79884975-9bf8-48b4-89c1-95e4f33e8285-kube-api-access-6c5tr\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.776346 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/368c881a-7e15-4fad-8acf-2253c547cc54-crio-socket\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.776565 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776364    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79884975-9bf8-48b4-89c1-95e4f33e8285-config-volume\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.776565 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/368c881a-7e15-4fad-8acf-2253c547cc54-data-volume\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.776565 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79884975-9bf8-48b4-89c1-95e4f33e8285-metrics-tls\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.776565 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxw7\" (UniqueName: \"kubernetes.io/projected/8d5029cc-b18a-4547-9e9e-7a172b56e789-kube-api-access-xdxw7\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.776941 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776916    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/368c881a-7e15-4fad-8acf-2253c547cc54-crio-socket\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.777142 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.777121    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/368c881a-7e15-4fad-8acf-2253c547cc54-data-volume\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.777278 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.776547    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79884975-9bf8-48b4-89c1-95e4f33e8285-tmp-dir\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.778094 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.778068    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/368c881a-7e15-4fad-8acf-2253c547cc54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.778225 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.778205    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79884975-9bf8-48b4-89c1-95e4f33e8285-config-volume\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.780942 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.780919    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79884975-9bf8-48b4-89c1-95e4f33e8285-metrics-tls\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.781391 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.781107    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d5029cc-b18a-4547-9e9e-7a172b56e789-cert\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.782851 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.782831    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/368c881a-7e15-4fad-8acf-2253c547cc54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.801815 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.801794    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fm4\" (UniqueName: \"kubernetes.io/projected/368c881a-7e15-4fad-8acf-2253c547cc54-kube-api-access-b4fm4\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.801956 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.801934    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5tr\" (UniqueName: \"kubernetes.io/projected/79884975-9bf8-48b4-89c1-95e4f33e8285-kube-api-access-6c5tr\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:33.802067 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.802053    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxw7\" (UniqueName: \"kubernetes.io/projected/8d5029cc-b18a-4547-9e9e-7a172b56e789-kube-api-access-xdxw7\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.824863 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.824831    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:33.828088 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.828072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:33.828139 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.828100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:33.828139 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.828099    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:33.833244 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.833225    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:33.844657 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.844560    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dvjrv\""
Oct 29 22:42:33.844657 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.844605    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:42:33.844657 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.844637    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:42:33.844657 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.844649    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgvrg\""
Oct 29 22:42:33.844876 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.844662    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrmx7\""
Oct 29 22:42:33.844876 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.844649    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:42:33.844876 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.844648    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:42:33.863514 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:33.863481    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:34.149987 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:34.149965    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7xwpt"]
Oct 29 22:42:34.152051 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:34.152024    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368c881a_7e15_4fad_8acf_2253c547cc54.slice/crio-b5ddd1f1a869731c32c4b836b5e6c2307de19e481ce488a41bf40666d14cd482 WatchSource:0}: Error finding container b5ddd1f1a869731c32c4b836b5e6c2307de19e481ce488a41bf40666d14cd482: Status 404 returned error can't find the container with id b5ddd1f1a869731c32c4b836b5e6c2307de19e481ce488a41bf40666d14cd482
Oct 29 22:42:34.156532 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:34.156512    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f67rh"]
Oct 29 22:42:34.157677 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:34.157653    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5029cc_b18a_4547_9e9e_7a172b56e789.slice/crio-1af74a9d33d2c3d17e82e788dd3d6036b873a970924c4dd44e584b3821cd7a1c WatchSource:0}: Error finding container 1af74a9d33d2c3d17e82e788dd3d6036b873a970924c4dd44e584b3821cd7a1c: Status 404 returned error can't find the container with id 1af74a9d33d2c3d17e82e788dd3d6036b873a970924c4dd44e584b3821cd7a1c
Oct 29 22:42:34.165386 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:34.165346    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xwpt" event={"ID":"368c881a-7e15-4fad-8acf-2253c547cc54","Type":"ContainerStarted","Data":"b5ddd1f1a869731c32c4b836b5e6c2307de19e481ce488a41bf40666d14cd482"}
Oct 29 22:42:34.166528 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:34.166502    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f67rh" event={"ID":"8d5029cc-b18a-4547-9e9e-7a172b56e789","Type":"ContainerStarted","Data":"1af74a9d33d2c3d17e82e788dd3d6036b873a970924c4dd44e584b3821cd7a1c"}
Oct 29 22:42:34.175212 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:34.175194    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqfnh"]
Oct 29 22:42:34.186617 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:34.186584    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79884975_9bf8_48b4_89c1_95e4f33e8285.slice/crio-2d5f554b365112328c67e6db17536ad41dbcbb9a9414437e652dbb2616a2687e WatchSource:0}: Error finding container 2d5f554b365112328c67e6db17536ad41dbcbb9a9414437e652dbb2616a2687e: Status 404 returned error can't find the container with id 2d5f554b365112328c67e6db17536ad41dbcbb9a9414437e652dbb2616a2687e
Oct 29 22:42:35.171833 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:35.171795    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqfnh" event={"ID":"79884975-9bf8-48b4-89c1-95e4f33e8285","Type":"ContainerStarted","Data":"2d5f554b365112328c67e6db17536ad41dbcbb9a9414437e652dbb2616a2687e"}
Oct 29 22:42:35.173607 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:35.173585    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xwpt" event={"ID":"368c881a-7e15-4fad-8acf-2253c547cc54","Type":"ContainerStarted","Data":"6d29328fdb091d6c98ae1cda882c048de656c9325aadd7a0d8bef39024afdab6"}
Oct 29 22:42:36.178627 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:36.178580    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xwpt" event={"ID":"368c881a-7e15-4fad-8acf-2253c547cc54","Type":"ContainerStarted","Data":"ebcbab4bd32a534df1e34343f50b09af7c744c1ef23357fbcd8a3227cea01816"}
Oct 29 22:42:37.183833 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:37.183797    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqfnh" event={"ID":"79884975-9bf8-48b4-89c1-95e4f33e8285","Type":"ContainerStarted","Data":"9e7dc213224e1041d40ce1bdbaadf3a04f3771657aac01d599f6df2af2a8bb59"}
Oct 29 22:42:37.184245 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:37.183851    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqfnh" event={"ID":"79884975-9bf8-48b4-89c1-95e4f33e8285","Type":"ContainerStarted","Data":"f1f920558504f18be59b0281a4a321f1933f3f7106121a9f51e512d5e1f6fe88"}
Oct 29 22:42:37.184245 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:37.183880    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:37.229690 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:37.229646    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vqfnh" podStartSLOduration=2.172304587 podStartE2EDuration="4.229633069s" totalImagesPullingTime="2.057328482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:37.228550906 +0000 UTC m=+33.025318039" watchObservedRunningTime="2026-10-29 22:42:37.229633069 +0000 UTC m=+33.026400199"
Oct 29 22:42:38.189335 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.189115    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f67rh" event={"ID":"8d5029cc-b18a-4547-9e9e-7a172b56e789","Type":"ContainerStarted","Data":"3676a42e638ad47acf750cb03a12c7bc9027e31b3e6f6a74f196d43cdcebccd1"}
Oct 29 22:42:38.191072 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.191044    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xwpt" event={"ID":"368c881a-7e15-4fad-8acf-2253c547cc54","Type":"ContainerStarted","Data":"e485567baa6a13e35743a2c7ef5e802efa88186edc5610ddd021783d94871ed8"}
Oct 29 22:42:38.233396 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.233357    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f67rh" podStartSLOduration=2.041771675 podStartE2EDuration="5.233346225s" totalImagesPullingTime="3.19157455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:38.233232217 +0000 UTC m=+34.029999350" watchObservedRunningTime="2026-10-29 22:42:38.233346225 +0000 UTC m=+34.030113357"
Oct 29 22:42:38.278555 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.278511    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7xwpt" podStartSLOduration=2.258186482 podStartE2EDuration="5.278497079s" totalImagesPullingTime="3.020310597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:38.278086637 +0000 UTC m=+34.074853775" watchObservedRunningTime="2026-10-29 22:42:38.278497079 +0000 UTC m=+34.075264211"
Oct 29 22:42:38.513246 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.513166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:38.513246 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.513208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:38.516969 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.516944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:38.516969 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.516956    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:38.614374 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.614342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:38.616746 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.616719    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:38.646687 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.646659    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:38.655819 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.655791    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:38.657880 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:38.657860    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:39.023422 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:39.023385    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-649bp"]
Oct 29 22:42:39.025294 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:39.025261    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e7869c_ca3a_4a1c_abac_f3bfa1639076.slice/crio-8b091d84b97c944b098f6fdda5be68dd3033a87ab28ebc849ab6ab658763ae75 WatchSource:0}: Error finding container 8b091d84b97c944b098f6fdda5be68dd3033a87ab28ebc849ab6ab658763ae75: Status 404 returned error can't find the container with id 8b091d84b97c944b098f6fdda5be68dd3033a87ab28ebc849ab6ab658763ae75
Oct 29 22:42:39.046487 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:39.046462    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nfc6s"]
Oct 29 22:42:39.047756 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:39.047733    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecd3290_1b8d_45ef_8381_b2aec149f522.slice/crio-3a917b7e47884ba3ae9a6e2ffcafcb6e3ea7c7bd180fc0a2e5ae1e5e677f9237 WatchSource:0}: Error finding container 3a917b7e47884ba3ae9a6e2ffcafcb6e3ea7c7bd180fc0a2e5ae1e5e677f9237: Status 404 returned error can't find the container with id 3a917b7e47884ba3ae9a6e2ffcafcb6e3ea7c7bd180fc0a2e5ae1e5e677f9237
Oct 29 22:42:39.060138 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:39.060115    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nt7zn"]
Oct 29 22:42:39.081886 ip-10-0-134-205 kubenswrapper[2574]: W1029 22:42:39.081866    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34812142_44fe_4a32_b03e_97fc0a461bd5.slice/crio-c9cd3ba4eb8b6aa3e6fcf59e35fcdbc39b9814fe81f4d54beeb13cb2cd4cf09c WatchSource:0}: Error finding container c9cd3ba4eb8b6aa3e6fcf59e35fcdbc39b9814fe81f4d54beeb13cb2cd4cf09c: Status 404 returned error can't find the container with id c9cd3ba4eb8b6aa3e6fcf59e35fcdbc39b9814fe81f4d54beeb13cb2cd4cf09c
Oct 29 22:42:39.194583 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:39.194549    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nt7zn" event={"ID":"34812142-44fe-4a32-b03e-97fc0a461bd5","Type":"ContainerStarted","Data":"c9cd3ba4eb8b6aa3e6fcf59e35fcdbc39b9814fe81f4d54beeb13cb2cd4cf09c"}
Oct 29 22:42:39.195673 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:39.195651    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-649bp" event={"ID":"e0e7869c-ca3a-4a1c-abac-f3bfa1639076","Type":"ContainerStarted","Data":"8b091d84b97c944b098f6fdda5be68dd3033a87ab28ebc849ab6ab658763ae75"}
Oct 29 22:42:39.196644 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:39.196622    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfc6s" event={"ID":"becd3290-1b8d-45ef-8381-b2aec149f522","Type":"ContainerStarted","Data":"3a917b7e47884ba3ae9a6e2ffcafcb6e3ea7c7bd180fc0a2e5ae1e5e677f9237"}
Oct 29 22:42:41.206976 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:41.206708    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfc6s" event={"ID":"becd3290-1b8d-45ef-8381-b2aec149f522","Type":"ContainerStarted","Data":"20ed2ca32ae80bd78af1fef6b723f8c7bbb6fe6ad6ca9577ab0490b4bde0d773"}
Oct 29 22:42:41.207438 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:41.207008    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfc6s" event={"ID":"becd3290-1b8d-45ef-8381-b2aec149f522","Type":"ContainerStarted","Data":"043455bd5c8e0d2564f012326d52d356fa9ca5cd2e07340a3f7b107ce7b6543f"}
Oct 29 22:42:41.251906 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:41.251849    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nfc6s" podStartSLOduration=34.920988842 podStartE2EDuration="36.251830236s" totalImagesPullingTime="1.330841394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:41.251233277 +0000 UTC m=+37.048000413" watchObservedRunningTime="2026-10-29 22:42:41.251830236 +0000 UTC m=+37.048597369"
Oct 29 22:42:43.216156 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:43.216063    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-649bp" event={"ID":"e0e7869c-ca3a-4a1c-abac-f3bfa1639076","Type":"ContainerStarted","Data":"718780f2faabb577c96b457251b81f6a48be503b44f58836f72b1e4cde8849ad"}
Oct 29 22:42:43.216554 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:43.216217    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:44.221257 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:44.221225    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nt7zn" event={"ID":"34812142-44fe-4a32-b03e-97fc0a461bd5","Type":"ContainerStarted","Data":"e96665e87a8824c8c01665c8fa03ecb1f486b7b8ee5d59fc1b4a72049866bf83"}
Oct 29 22:42:44.263526 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:44.263442    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-649bp" podStartSLOduration=35.464873588 podStartE2EDuration="39.263427458s" totalImagesPullingTime="3.79855387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:43.259077068 +0000 UTC m=+39.055844200" watchObservedRunningTime="2026-10-29 22:42:44.263427458 +0000 UTC m=+40.060194590"
Oct 29 22:42:44.263826 ip-10-0-134-205 kubenswrapper[2574]: I1029 22:42:44.263803    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nt7zn" podStartSLOduration=34.352264777 podStartE2EDuration="39.263797679s" totalImagesPullingTime="4.911532902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:44.263229875 +0000 UTC m=+40.059997008" watchObservedRunningTime="2026-10-29 22:42:44.263797679 +0000 UTC m=+40.060564811"
Oct 29 22:42:44.286346 ip-10-0-134-205 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 22:42:44.308387 ip-10-0-134-205 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 22:42:44.308554 ip-10-0-134-205 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 22:42:44.308718 ip-10-0-134-205 systemd[1]: kubelet.service: Consumed 2.466s CPU time.
Oct 29 22:42:44.319342 ip-10-0-134-205 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:42:44.406233 ip-10-0-134-205 kubenswrapper[4827]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:44.406233 ip-10-0-134-205 kubenswrapper[4827]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:42:44.406233 ip-10-0-134-205 kubenswrapper[4827]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:44.406233 ip-10-0-134-205 kubenswrapper[4827]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:44.408328 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408250    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.408328 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408328    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408332    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408336    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408339    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408342    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408344    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408347    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408359    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408362    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408365    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408368    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408372    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408376    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408379    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408382    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408384    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408387    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408390    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408393    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.408398 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408395    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408398    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408401    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408405    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408408    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408410    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408413    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408416    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408419    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408421    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408424    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408426    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408429    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408432    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408434    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408437    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408439    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408442    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408445    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408448    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.408891 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408451    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408453    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408456    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408458    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408461    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408463    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408466    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408469    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408471    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408474    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408477    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408480    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408482    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408485    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408488    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408491    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408493    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408496    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408498    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.409364 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408501    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408503    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408506    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408509    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408511    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408514    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408516    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408519    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408521    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408524    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408526    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408529    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408532    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408535    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408544    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408547    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408550    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408552    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408555    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408558    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.409957 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408561    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408563    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408566    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408568    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408571    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408574    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408576    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408579    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408581    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408584    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408586    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408589    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408592    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408594    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408597    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408599    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408602    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408604    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408607    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408609    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.410434 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408612    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408633    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408638    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408642    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408647    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408650    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.408653    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409399    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409406    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409409    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409412    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409428    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409433    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409436    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409439    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409442    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409444    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409447    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409450    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.410982 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409452    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409455    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409457    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409460    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409462    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409465    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409468    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409471    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409473    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409476    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409480    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409484    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409487    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409491    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409495    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409498    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409501    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409506    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409509    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.411452 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409512    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409520    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409523    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409525    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409528    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409531    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409534    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409536    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409539    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409541    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409544    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409546    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409549    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409551    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409554    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409557    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409559    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409562    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409564    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409567    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.411937 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409570    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409572    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409575    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409578    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409580    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409583    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409585    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409594    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409597    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409599    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409603    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409606    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409608    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409611    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409619    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409622    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409625    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409628    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409630    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.412442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409633    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409635    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409638    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409640    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409643    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409645    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409648    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409650    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409653    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409655    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409658    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409660    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409663    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409665    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409668    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409670    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409673    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409676    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409679    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409681    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.412928 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409684    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409686    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409689    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409693    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409696    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409699    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409701    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409704    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409707    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409709    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409712    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409714    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409716    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409719    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409722    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.409724    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410219    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410224    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410227    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410230    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.413446 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410232    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410236    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410238    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410241    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410244    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410247    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410249    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410252    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410254    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410257    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410260    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410262    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410265    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410267    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410270    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410274    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410277    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410281    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410285    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.413944 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410288    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410292    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410295    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410299    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410301    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410304    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410307    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410310    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410313    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410317    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410319    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410322    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410325    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410327    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410330    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410332    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410335    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410337    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410340    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.414417 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410343    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410346    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410349    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410351    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410354    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410356    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410359    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410361    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410364    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410367    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410370    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410373    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410375    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410378    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410380    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410383    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410386    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410389    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410391    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.414906 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410394    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410396    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410399    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410402    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410405    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410408    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410410    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410413    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410416    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410418    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410421    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410423    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410426    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410428    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410431    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410434    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410436    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410439    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410441    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.415511 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410445    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410447    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410451    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410455    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410457    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410460    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410463    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410465    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410468    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410471    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410474    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410476    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410479    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410481    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410484    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410486    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410490    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410493    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410495    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410498    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.416262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410500    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410504    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410507    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410509    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410512    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.410515    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410604    4827 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410611    4827 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410617    4827 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410622    4827 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410626    4827 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410629    4827 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410634    4827 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410638    4827 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410641    4827 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410646    4827 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410649    4827 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410653    4827 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410656    4827 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410659    4827 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410661    4827 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410665    4827 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410668    4827 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:42:44.416782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410671    4827 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410676    4827 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410679    4827 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410682    4827 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410685    4827 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410689    4827 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410693    4827 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410696    4827 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410699    4827 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410703    4827 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410706    4827 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410709    4827 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410712    4827 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410715    4827 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410718    4827 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410722    4827 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410725    4827 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410728    4827 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410731    4827 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410734    4827 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410736    4827 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410740    4827 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410743    4827 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410746    4827 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:42:44.417339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410751    4827 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410754    4827 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410758    4827 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410761    4827 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410778    4827 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410781    4827 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410785    4827 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410788    4827 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410791    4827 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410794    4827 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410797    4827 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410800    4827 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410803    4827 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410809    4827 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410812    4827 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410815    4827 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410818    4827 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410821    4827 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410824    4827 flags.go:64] FLAG: --help="false"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410827    4827 flags.go:64] FLAG: --hostname-override="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410830    4827 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410833    4827 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410836    4827 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410840    4827 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:42:44.417984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410843    4827 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410846    4827 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410849    4827 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410852    4827 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410854    4827 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410857    4827 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410860    4827 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410863    4827 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410867    4827 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410870    4827 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410873    4827 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410876    4827 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410879    4827 flags.go:64] FLAG: --lock-file=""
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410881    4827 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410884    4827 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410887    4827 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410893    4827 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410896    4827 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410899    4827 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410902    4827 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410905    4827 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410909    4827 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410912    4827 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410915    4827 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410920    4827 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:42:44.418578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410923    4827 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410926    4827 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410929    4827 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410932    4827 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410935    4827 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410938    4827 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410941    4827 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410944    4827 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410947    4827 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410954    4827 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410957    4827 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410960    4827 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410963    4827 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410966    4827 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410969    4827 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410973    4827 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410976    4827 flags.go:64] FLAG: --port="10250"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410979    4827 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410983    4827 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c3d01b4618fa85d2"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410986    4827 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410989    4827 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410993    4827 flags.go:64] FLAG: --register-node="true"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.410995    4827 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411000    4827 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411003    4827 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411006    4827 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:42:44.419415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411009    4827 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411013    4827 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411017    4827 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411020    4827 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411023    4827 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411026    4827 flags.go:64] FLAG: --runonce="false"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411029    4827 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411032    4827 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411035    4827 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411038    4827 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411041    4827 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411044    4827 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411048    4827 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411050    4827 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411053    4827 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411056    4827 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411059    4827 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411062    4827 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411065    4827 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411068    4827 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411071    4827 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411077    4827 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411080    4827 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411082    4827 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411087    4827 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:42:44.420079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411089    4827 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411092    4827 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411095    4827 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411098    4827 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411101    4827 flags.go:64] FLAG: --v="2"
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411106    4827 flags.go:64] FLAG: --version="false"
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411110    4827 flags.go:64] FLAG: --vmodule=""
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411114    4827 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411117    4827 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411251    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411255    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411258    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411261    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411264    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411267    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411269    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411272    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411274    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411277    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411279    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411282    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.420693 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411285    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411288    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411290    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411293    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411295    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411299    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411303    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411306    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411309    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411312    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411315    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411318    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411320    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411323    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411326    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411328    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411331    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411334    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411336    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411339    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.421312 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411343    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411346    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411348    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411351    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411353    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411356    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411359    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411361    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411364    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411366    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411369    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411371    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411374    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411376    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411379    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411381    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411384    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411386    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411389    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411391    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.422043 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411394    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411397    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411399    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411403    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411407    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411411    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411414    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411416    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411419    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411422    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411425    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411427    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411431    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411434    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411436    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411439    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411441    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411444    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411447    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411449    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.422679 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411452    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411454    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411457    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411460    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411463    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411466    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411468    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411471    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411482    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411485    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411488    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411490    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411493    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411496    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411498    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411501    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411503    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411506    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411508    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.423362 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411511    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411513    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411516    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411518    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411521    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411526    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411529    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411532    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411534    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411537    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411540    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411542    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411545    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411547    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411550    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.411576    4827 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:44.423854 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411627    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411632    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411635    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411638    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411641    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411643    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411646    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411649    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411651    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411654    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411657    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411659    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411662    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411664    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411667    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411670    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411672    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411675    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411679    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.424239 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411683    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411687    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411691    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411696    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411699    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411702    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411705    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411708    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411711    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411713    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411716    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411719    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411722    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411725    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411727    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411730    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411733    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411735    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411738    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411740    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.424967 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411743    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411746    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411748    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411751    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411753    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411756    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411759    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411773    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411776    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411778    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411781    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411784    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411786    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411789    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411793    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411797    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411800    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411802    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411810    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.425579 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411813    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411816    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411818    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411821    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411824    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411826    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411829    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411831    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411834    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411837    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411840    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411842    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411845    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411847    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411850    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411852    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411855    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411858    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411860    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.426066 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411863    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411865    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411868    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411870    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411873    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411875    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411878    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411880    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411884    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411889    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411893    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411896    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411899    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411902    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411905    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411907    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411910    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411913    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411916    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411918    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.426559 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411921    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411923    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411926    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411928    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411931    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411934    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411937    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411940    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.411943    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.417264    4827 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.417281    4827 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.417304    4827 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.417309    4827 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417367    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417373    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417376    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417379    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417381    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417384    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.427048 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417387    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417390    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417393    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417395    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417398    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417401    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417404    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417407    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417410    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417413    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417416    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417419    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417421    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417424    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417426    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417429    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417432    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417434    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417437    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417439    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.427506 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417442    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417444    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417448    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417451    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417453    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417457    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417459    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417462    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417464    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417467    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417470    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417472    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417475    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417478    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417480    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417483    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417485    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417488    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417490    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417493    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.428001 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417496    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417498    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417501    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417503    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417506    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417508    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417511    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417514    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417516    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417519    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417521    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417524    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417526    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417529    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417532    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417535    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417537    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417540    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417543    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.428468 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417546    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417549    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417551    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417554    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417556    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417559    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417561    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417564    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417567    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417570    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417574    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417577    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417580    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417583    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417586    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417589    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417592    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417594    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417597    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417599    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.428945 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417602    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417605    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417607    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417612    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417616    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417619    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417622    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417625    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417628    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417631    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417634    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417636    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417639    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417641    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417644    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417646    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417649    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417651    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417654    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417656    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.429742 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417659    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.417687    4827 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417841    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417847    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417850    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417853    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417857    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417861    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417864    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417867    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417869    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417872    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417875    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417877    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417880    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417882    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.430461 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417885    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417888    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417892    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417896    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417899    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417902    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417904    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417907    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417910    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417912    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417915    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417917    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417920    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417923    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417925    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417927    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417930    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417933    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417935    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.430871 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417938    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417940    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417943    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417945    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417948    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417951    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417954    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417956    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417959    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417962    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417964    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417967    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417969    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417972    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417974    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417977    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417979    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417982    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417985    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417988    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.431315 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417991    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417993    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417996    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.417999    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418002    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418004    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418007    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418009    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418012    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418014    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418017    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418019    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418021    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418024    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418026    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418029    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418031    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418034    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418037    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.431808 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418040    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418042    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418045    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418048    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418050    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418053    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418055    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418058    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418060    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418063    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418065    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418067    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418070    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418073    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418076    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418078    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418081    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418084    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418086    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418089    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.432293 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418091    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418094    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418097    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418099    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418102    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418104    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418107    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418109    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418112    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418114    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418117    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418119    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418122    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418125    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.418148    4827 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418179    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418183    4827 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:44.432866 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418186    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418189    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418192    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418195    4827 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418197    4827 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418200    4827 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418203    4827 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418205    4827 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418208    4827 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418211    4827 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418213    4827 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418216    4827 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418218    4827 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418221    4827 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418223    4827 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418226    4827 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418228    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418231    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418235    4827 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418238    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:44.433317 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418241    4827 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418244    4827 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418247    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418250    4827 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418252    4827 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418255    4827 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418258    4827 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418260    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418263    4827 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418266    4827 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418270    4827 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418273    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418277    4827 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418279    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418283    4827 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418285    4827 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418288    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418291    4827 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418293    4827 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:44.433816 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418296    4827 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418298    4827 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418304    4827 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418306    4827 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418309    4827 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418312    4827 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418314    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418317    4827 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418320    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418322    4827 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418325    4827 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418327    4827 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418330    4827 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418333    4827 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418335    4827 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418337    4827 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418340    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418342    4827 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418345    4827 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:44.434262 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418348    4827 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418350    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418353    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418356    4827 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418358    4827 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418361    4827 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418363    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418366    4827 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418369    4827 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418371    4827 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418374    4827 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418376    4827 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418379    4827 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418381    4827 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418384    4827 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418388    4827 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418390    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418393    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418395    4827 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:44.434855 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418398    4827 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418400    4827 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418403    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418405    4827 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418408    4827 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418411    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418413    4827 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418416    4827 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418418    4827 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418421    4827 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418423    4827 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418426    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418428    4827 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418431    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418433    4827 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418436    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418438    4827 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418441    4827 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418444    4827 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:44.435619 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418446    4827 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418449    4827 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418452    4827 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418454    4827 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418457    4827 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418460    4827 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418463    4827 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:42:44.418465    4827 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.418521    4827 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.419469    4827 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.419523    4827 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.419689    4827 server.go:1014] "Starting client certificate rotation"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.419786    4827 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.419825    4827 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:37:04 +0000 UTC" deadline="2028-05-28 02:14:33.663705241 +0000 UTC"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.419857    4827 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13827h31m49.243850317s"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.420151    4827 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:44.436359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.421141    4827 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:44.437105 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.422653    4827 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:42:44.437105 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.424340    4827 log.go:25] "Validated CRI v1 image API"
Oct 29 22:42:44.437105 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.425125    4827 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:42:44.437105 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.426728    4827 fs.go:135] Filesystem UUIDs: map[3490f51c-f590-4daa-b0c5-67ce535ddea9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b28cd44d-ab8d-47b0-8e93-14356514a3a9:/dev/nvme0n1p4]
Oct 29 22:42:44.437234 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.426748    4827 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b196787d199bfdb333da3fb04debaef4400bb6cde6b235f38c117b78b1c2630/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b196787d199bfdb333da3fb04debaef4400bb6cde6b235f38c117b78b1c2630/userdata/shm major:0 minor:99 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0da6962a6847d0f181a2713c35b65d61958ba76ec442c4577b279434dc3344b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0da6962a6847d0f181a2713c35b65d61958ba76ec442c4577b279434dc3344b9/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1af74a9d33d2c3d17e82e788dd3d6036b873a970924c4dd44e584b3821cd7a1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1af74a9d33d2c3d17e82e788dd3d6036b873a970924c4dd44e584b3821cd7a1c/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d5f554b365112328c67e6db17536ad41dbcbb9a9414437e652dbb2616a2687e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d5f554b365112328c67e6db17536ad41dbcbb9a9414437e652dbb2616a2687e/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/378dfaabbdb187afb5199200e0dd36d222d53a287e3723a79b3f838f25e767ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/378dfaabbdb187afb5199200e0dd36d222d53a287e3723a79b3f838f25e767ba/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a917b7e47884ba3ae9a6e2ffcafcb6e3ea7c7bd180fc0a2e5ae1e5e677f9237/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a917b7e47884ba3ae9a6e2ffcafcb6e3ea7c7bd180fc0a2e5ae1e5e677f9237/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/796ee053bd9e33bfbe6c068f283ddbe307a4af4a1083a198c432116098756199/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/796ee053bd9e33bfbe6c068f283ddbe307a4af4a1083a198c432116098756199/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79d3f2b99ef3f097f6215c4b1af730b6990fdf6a7bb744610529a34c9ef05e41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79d3f2b99ef3f097f6215c4b1af730b6990fdf6a7bb744610529a34c9ef05e41/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b091d84b97c944b098f6fdda5be68dd3033a87ab28ebc849ab6ab658763ae75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b091d84b97c944b098f6fdda5be68dd3033a87ab28ebc849ab6ab658763ae75/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8da461694f00fde0c4bb4751c27a73b36796e93809e583c1d566923daad2f04b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8da461694f00fde0c4bb4751c27a73b36796e93809e583c1d566923daad2f04b/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2b3b8b11d76e465cbb6de9a72c6da60f625ff03a98ae4e40b85823e7fda8939/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2b3b8b11d76e465cbb6de9a72c6da60f625ff03a98ae4e40b85823e7fda8939/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5ddd1f1a869731c32c4b836b5e6c2307de19e481ce488a41bf40666d14cd482/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5ddd1f1a869731c32c4b836b5e6c2307de19e481ce488a41bf40666d14cd482/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8d946a20c0900abd0a1d268bc63105e0f1c69ed7f9c1285da30aa1bfe2b8503/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8d946a20c0900abd0a1d268bc63105e0f1c69ed7f9c1285da30aa1bfe2b8503/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9cd3ba4eb8b6aa3e6fcf59e35fcdbc39b9814fe81f4d54beeb13cb2cd4cf09c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9cd3ba4eb8b6aa3e6fcf59e35fcdbc39b9814fe81f4d54beeb13cb2cd4cf09c/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6e3908315ccab466925439ffe555999a3b7490ae2c51847dea467ab87315baa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6e3908315ccab466925439ffe555999a3b7490ae2c51847dea467ab87315baa/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13e0ca11-32d7-4d64-9388-7f50197dd515/volumes/kubernetes.io~projected/kube-api-access-6pskr:{mountpoint:/var/lib/kubelet/pods/13e0ca11-32d7-4d64-9388-7f50197dd515/volumes/kubernetes.io~projected/kube-api-access-6pskr major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34812142-44fe-4a32-b03e-97fc0a461bd5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/34812142-44fe-4a32-b03e-97fc0a461bd5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/368c881a-7e15-4fad-8acf-2253c547cc54/volumes/kubernetes.io~projected/kube-api-access-b4fm4:{mountpoint:/var/lib/kubelet/pods/368c881a-7e15-4fad-8acf-2253c547cc54/volumes/kubernetes.io~projected/kube-api-access-b4fm4 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/368c881a-7e15-4fad-8acf-2253c547cc54/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/368c881a-7e15-4fad-8acf-2253c547cc54/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5612c1b5-6517-4a68-a0a4-3a10ff070d31/volumes/kubernetes.io~projected/kube-api-access-slmrs:{mountpoint:/var/lib/kubelet/pods/5612c1b5-6517-4a68-a0a4-3a10ff070d31/volumes/kubernetes.io~projected/kube-api-access-slmrs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79884975-9bf8-48b4-89c1-95e4f33e8285/volumes/kubernetes.io~projected/kube-api-access-6c5tr:{mountpoint:/var/lib/kubelet/pods/79884975-9bf8-48b4-89c1-95e4f33e8285/volumes/kubernetes.io~projected/kube-api-access-6c5tr major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79884975-9bf8-48b4-89c1-95e4f33e8285/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/79884975-9bf8-48b4-89c1-95e4f33e8285/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5029cc-b18a-4547-9e9e-7a172b56e789/volumes/kubernetes.io~projected/kube-api-access-xdxw7:{mountpoint:/var/lib/kubelet/pods/8d5029cc-b18a-4547-9e9e-7a172b56e789/volumes/kubernetes.io~projected/kube-api-access-xdxw7 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5029cc-b18a-4547-9e9e-7a172b56e789/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8d5029cc-b18a-4547-9e9e-7a172b56e789/volumes/kubernetes.io~secret/cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3a24a6b-c505-47ea-9400-f359710c3883/volumes/kubernetes.io~projected/kube-api-access-n5vct:{mountpoint:/var/lib/kubelet/pods/b3a24a6b-c505-47ea-9400-f359710c3883/volumes/kubernetes.io~projected/kube-api-access-n5vct major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5caa8e9-a021-4107-8fb3-945c7b1179e6/volumes/kubernetes.io~projected/kube-api-access-xd7fl:{mountpoint:/var/lib/kubelet/pods/b5caa8e9-a021-4107-8fb3-945c7b1179e6/volumes/kubernetes.io~projected/kube-api-access-xd7fl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~projected/kube-api-access-frmnh:{mountpoint:/var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~projected/kube-api-access-frmnh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/becd3290-1b8d-45ef-8381-b2aec149f522/volumes/kubernetes.io~projected/kube-api-access-vp9pw:{mountpoint:/var/lib/kubelet/pods/becd3290-1b8d-45ef-8381-b2aec149f522/volumes/kubernetes.io~projected/kube-api-access-vp9pw major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/becd3290-1b8d-45ef-8381-b2aec149f522/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/becd3290-1b8d-45ef-8381-b2aec149f522/volumes/kubernetes.io~secret/metrics-certs major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8ea7bf0-7270-447b-8aeb-a91164efe631/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c8ea7bf0-7270-447b-8aeb-a91164efe631/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~projected/kube-api-access-z5pbs:{mountpoint:/var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~projected/kube-api-access-z5pbs major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1014721-61ff-4d6e-b0f7-83b4f51649c5/volumes/kubernetes.io~projected/kube-api-access-9lrnj:{mountpoint:/var/lib/kubelet/pods/d1014721-61ff-4d6e-b0f7-83b4f51649c5/volumes/kubernetes.io~projected/kube-api-access-9lrnj major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4b0b131-a728-4cee-aa9e-efa80695c400/volumes/kubernetes.io~projected/kube-api-access-qlzvq:{mountpoint:/var/lib/kubelet/pods/d4b0b131-a728-4cee-aa9e-efa80695c400/volumes/kubernetes.io~projected/kube-api-access-qlzvq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volumes/kubernetes.io~projected/kube-api-access-64s69:{mountpoint:/var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volumes/kubernetes.io~projected/kube-api-access-64s69 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0e7869c-ca3a-4a1c-abac-f3bfa1639076/volumes/kubernetes.io~projected/kube-api-access-5qvth:{mountpoint:/var/lib/kubelet/pods/e0e7869c-ca3a-4a1c-abac-f3bfa1639076/volumes/kubernetes.io~projected/kube-api-access-5qvth major:0 minor:269 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-101:{mountpoint:/var/lib/containers/storage/overlay/1c9406776e1c62e8be4f5f4d4d81a59a5d24fce8546c5abb012ccc8ac19b77a2/merged major:0 minor:101 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/75bb5377c77cd14b5ea6abb649bc446b7c226fdee84c2650d2418d96c7dc89f8/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/57ec31d6cc0563d81238171135cdd373de697570039213830cca97da26aed1d1/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/4ef648d99267111acca6551f740b94089ce2001e5291dab2394eac342efbaacb/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/f5b62968d731e178203ba4923ae6309a1535d95dba98b99151bef50cd13e6b36/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/d1c647d80d476edf849ea5ea82e158607704dd19ec4413dc9b95ccab51b43d7b/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/f0f079df6b32545ff71f548e843ce0dcb41e0c952a21439c86a09b9e6efee000/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/containers/storage/overlay/57d9902000944d15047318166deb17bb89852f21f3b2c7bfc00962a1df339bed/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/0e4f2a6f5e0f1fdf4ff291f3520eb6c974da2125950b30f7b9b9b43d58ad5068/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/8d0ba9214ba969f69a16b988a8ebaadd8d805209111c7d6862c8d61386485184/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/4bdc80fba24fab4b7dc4addc99de6f29628b1ca63dfddadb3954195019352bcd/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/0770005f0427a61727f40e8695fe6bc32fd27f08238e1325eba9818003233170/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/b65887670ec7839610c83d67e2f82edcb46296b4238b377d197abc75bdf49246/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-150:{mountpoint:/var/lib/containers/storage/overlay/b68e514c0630157bc0fb2d456f5f90c4870d804eb764e9438093d5efa9329842/merged major:0 minor:150 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3a10545ab274adc80a7b21d9a52525e6943df2a0a7a830f6486af37446f3e34d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/bcd692ca2b33a4a01acfe662c4b5a3eff6880ad9723e674e6386b7272b9e8209/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/dd2c294746858ebf2f4f9f50f135b76aa3040026539069df97fb7394b7e3c69f/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/5909616d361ab84c0208eea6458949138cec971868db2abe8ee137dcfd7f3529/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/e590842585ae94efc9175872c1554902555a45e8d8f92be53db289d0503364c9/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/03ac68f527711908bcba8305c7a577dc6ca989337c336d76b542a947fd1a67c6/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/19b134731d96c4757ed4df34713f5f2337eb3768a7b1d0a873a43a2b9723015c/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/2b2fa90d4d0ec0e66a2404fe8c7cd055514ae22423346c36fc8204f56375563f/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-199:{mountpoint:/var/lib/containers/storage/overlay/84b1e370c5f26630adb74c733d8aa7d29738b28b32cc42f445d5eabcbb09b14e/merged major:0 minor:199 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/ad5dd986b72084419208504612e46eb2a1ed54e483ed0c27686a4a5ce024c135/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/5b0aa69bcf805af22b55d55a09b3b87beed422ee48d5cb7dbaa50b6c8818dbf0/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/dc261fb9f804abfd72e6945b6364d9f66fe83c5e6434f87bf05e9aab5b7b097f/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/eeef439dcdf329aad58753b64070b73510f0cebf0494b8c3b8e19e6042a54e3b/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/e2e6b5919ac1f4fdd1d1c4a247141f3e1f066a00daef1f14f0d91705f0d98c24/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/09adb70c194fb1354a1f041cbf21dc275c5f88921deb4ba1c3bc005e6825480a/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/124e4a428ae352d0b1336d22d801bec97699e66539e1b6423ae97e3183d82dbc/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/ddfc95acfcc34352bea6b1eadd2810fd3edea604f5b399b72d3e7fb36f99950e/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/13d113793c729792ea6ebccefc55162d68ca84b3323d9e631ae9dc0d3a72826a/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/e60c605b63b2addf83f041ac67c8ae7b0fc933310dd01511ae1142dac860d437/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/0fef7c8b9b07828ceabc0b556d773c4ffad51f5b7ea9cf0289dae8bb27446a07/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c75ab72b386ad7d486e97249941f1517057381e61e8418b80ac16542f3fa198a/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/12202a329813605faab713185e0cf6cad62c89cb7edad5d72d1adcdc9f6bc61b/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/7a37f6c749235e339908b5b522aec43d7f1fc5b5123ee5ae8a07366680d1a257/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/a8bfa024775ac9bef09db4161f981e91ea66c675bbcb700935b820c8e90a8163/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/49ad178fb1f111c2f6d64ec32a4b51635cdc41c6bbfe64b5f3ae1c5010b06c46/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/5809ea01dfe4db2396bbbcea5781a22f36d94450ab17be2fe421e81c17ccfb6b/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/2e91fa51c683a12a5686fe98c97d8db951b5319058fb57b9591ee03ee5508683/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/30f47e85ac37c71d879c384a7c416420fc51206e779e20dbee0aa2280e710695/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6852a7dc57cdd77d9a537e94b702a6de6b33bf5cac10114dfc5fce8d27f9181c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/8ae46c340b923e50b3687c6fdd4d0c0f667cb0cbf0e125f61e26cb64624227fa/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/159096213e59e445ccb85158647e21f2ae426e28bda2bb1574ca2245b2873f02/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/71556a659f07e3a69c509eb3e678e3f5bcd8639814b9c9653cb6b5a9ced32f26/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/5b43b5032ecccf04e34eeb5a99498f5777ea68c4e33929b1bf057dfcd5235c61/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/2a94221c6d857bee32cedc33ca80741000ee71453f3ac4323f52cba87a845d76/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/5c663bc7e2203faf12254876433736f121ded5ef5f22be66d2dc2c64cbdea56e/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/89c78b5f7ba2d301843863a901aa29588be9dfbdb82a7e1bd27d4783b0308bcf/merged major:0 minor:97 fsType:overlay blockSize:0}]
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.434124    4827 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:44.433619361 +0000 UTC m=+0.056530373 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099755 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec4f5cbdf26456910a5b873861821 SystemUUID:ec2ec4f5-cbdf-2645-6910-a5b873861821 BootID:995fd839-fb92-4016-98f3-1d22a369a05f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0da6962a6847d0f181a2713c35b65d61958ba76ec442c4577b279434dc3344b9/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/368c881a-7e15-4fad-8acf-2253c547cc54/volumes/kubernetes.io~projected/kube-api-access-b4fm4 DeviceMajor:0 DeviceMinor:220 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b3a24a6b-c505-47ea-9400-f359710c3883/volumes/kubernetes.io~projected/kube-api-access-n5vct DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/13e0ca11-32d7-4d64-9388-7f50197dd515/volumes/kubernetes.io~projected/kube-api-access-6pskr DeviceMajor:0 DeviceMinor:69 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0e7869c-ca3a-4a1c-abac-f3bfa1639076/volumes/kubernetes.io~projected/kube-api-access-5qvth DeviceMajor:0 DeviceMinor:269 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5029cc-b18a-4547-9e9e-7a172b56e789/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5ddd1f1a869731c32c4b836b5e6c2307de19e481ce488a41bf40666d14cd482/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1014721-61ff-4d6e-b0f7-83b4f51649c5/volumes/kubernetes.io~projected/kube-api-access-9lrnj DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b196787d199bfdb333da3fb04debaef4400bb6cde6b235f38c117b78b1c2630/userdata/shm DeviceMajor:0 DeviceMinor:99 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a917b7e47884ba3ae9a6e2ffcafcb6e3ea7c7bd180fc0a2e5ae1e5e677f9237/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volumes/kubernetes.io~projected/kube-api-access-64s69 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b091d84b97c944b098f6fdda5be68dd3033a87ab28ebc849ab6ab658763ae75/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c8ea7bf0-7270-447b-8aeb-a91164efe631/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8d946a20c0900abd0a1d268bc63105e0f1c69ed7f9c1285da30aa1bfe2b8503/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~projected/kube-api-access-z5pbs DeviceMajor:0 DeviceMinor:70 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/378dfaabbdb187afb5199200e0dd36d222d53a287e3723a79b3f838f25e767ba/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5029cc-b18a-4547-9e9e-7a172b56e789/volumes/kubernetes.io~projected/kube-api-access-xdxw7 DeviceMajor:0 DeviceMinor:221 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1af74a9d33d2c3d17e82e788dd3d6036b873a970924c4dd44e584b3821cd7a1c/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~projected/kube-api-access-frmnh DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2b3b8b11d76e465cbb6de9a72c6da60f625ff03a98ae4e40b85823e7fda8939/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6e3908315ccab466925439ffe555999a3b7490ae2c51847dea467ab87315baa/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d5f554b365112328c67e6db17536ad41dbcbb9a9414437e652dbb2616a2687e/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/becd3290-1b8d-45ef-8381-b2aec149f522/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:260 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/becd3290-1b8d-45ef-8381-b2aec149f522/volumes/kubernetes.io~projected/kube-api-access-vp9pw DeviceMajor:0 DeviceMinor:68 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9cd3ba4eb8b6aa3e6fcf59e35fcdbc39b9814fe81f4d54beeb13cb2cd4cf09c/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5612c1b5-6517-4a68-a0a4-3a10ff070d31/volumes/kubernetes.io~projected/kube-api-access-slmrs DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8da461694f00fde0c4bb4751c27a73b36796e93809e583c1d566923daad2f04b/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/368c881a-7e15-4fad-8acf-2253c547cc54/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34812142-44fe-4a32-b03e-97fc0a461bd5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:261 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5caa8e9-a021-4107-8fb3-945c7b1179e6/volumes/kubernetes.io~projected/kube-api-access-xd7fl DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-150 DeviceMajor:0 DeviceMinor:150 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79884975-9bf8-48b4-89c1-95e4f33e8285/volumes/kubernetes.io~projected/kube-api-access-6c5tr DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-199 DeviceMajor:0 DeviceMinor:199 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c9d9a88e-975d-4596-9c23-1144707d5391/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/be8bd6d3-88a1-482b-a514-d254705c5cbc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/796ee053bd9e33bfbe6c068f283ddbe307a4af4a1083a198c432116098756199/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79d3f2b99ef3f097f6215c4b1af730b6990fdf6a7bb744610529a34c9ef05e41/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4b0b131-a728-4cee-aa9e-efa80695c400/volumes/kubernetes.io~projected/kube-api-access-qlzvq DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc8ed141-dd58-45a2-841e-22c1d8eb19c1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/79884975-9bf8-48b4-89c1-95e4f33e8285/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1af74a9d33d2c3d MacAddress:e6:ba:1d:b1:2f:a7 Speed:10000 Mtu:8901} {Name:2d5f554b3651123 MacAddress:26:32:33:c6:f8:a8 Speed:10000 Mtu:8901} {Name:3a917b7e47884ba MacAddress:de:3f:c2:55:85:26 Speed:10000 Mtu:8901} {Name:8b091d84b97c944 MacAddress:ae:90:8d:95:20:5a Speed:10000 Mtu:8901} {Name:b5ddd1f1a869731 MacAddress:9a:eb:ff:17:23:68 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:fc:96:e7 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:ac:ad:19:df:dd Speed:0 Mtu:8901} {Name:c9cd3ba4eb8b6aa MacAddress:1e:a6:7f:4f:b7:17 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:fc:96:e7 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:0f:3b:ae:a3:ef Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:98:69:5b:02:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.434422    4827 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.434512    4827 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.434878    4827 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.434913    4827 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435132    4827 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435145    4827 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435158    4827 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435184    4827 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435293    4827 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435549    4827 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435690    4827 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435735    4827 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435749    4827 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435778    4827 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435791    4827 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.435802    4827 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.436528    4827 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.436539    4827 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:44.437406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.437107    4827 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:42:44.439126 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439113    4827 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:42:44.439126 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439126    4827 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:44.439238 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439228    4827 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 22:42:44.439416 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439406    4827 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:42:44.439444 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439434    4827 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:42:44.439573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439564    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:42:44.439603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439579    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:42:44.439603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439585    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:42:44.439603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439591    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:42:44.439603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439596    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:42:44.439603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439604    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439609    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439616    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439625    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439634    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439642    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439672    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439698    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:42:44.439734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.439707    4827 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:42:44.443031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.443018    4827 server.go:1263] "Started kubelet"
Oct 29 22:42:44.443128 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.443080    4827 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:42:44.443544 ip-10-0-134-205 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:42:44.444616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.443101    4827 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:42:44.444616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.443892    4827 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:42:44.444616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.444283    4827 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:42:44.444616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.444330    4827 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:42:44.447916 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.447897    4827 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:42:44.448012 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.447915    4827 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:42:44.448012 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.447989    4827 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:06 +0000 UTC" deadline="2028-05-23 11:59:53.12219154 +0000 UTC"
Oct 29 22:42:44.448182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.448021    4827 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13717h17m8.674174235s"
Oct 29 22:42:44.448182 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:44.448050    4827 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-205.ec2.internal\" not found"
Oct 29 22:42:44.448547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.448529    4827 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:42:44.450207 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.448022    4827 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:42:44.450207 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.450122    4827 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:42:44.450878 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.450860    4827 factory.go:153] Registering CRI-O factory
Oct 29 22:42:44.450957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.450882    4827 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:42:44.450957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.450950    4827 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:42:44.451025 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.450960    4827 factory.go:55] Registering systemd factory
Oct 29 22:42:44.451025 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.450969    4827 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:42:44.451025 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.450995    4827 factory.go:103] Registering Raw factory
Oct 29 22:42:44.451025 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.451008    4827 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:42:44.451358 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.451344    4827 manager.go:319] Starting recovery of all containers
Oct 29 22:42:44.453040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453006    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13e0ca11-32d7-4d64-9388-7f50197dd515" volumeName="kubernetes.io/projected/13e0ca11-32d7-4d64-9388-7f50197dd515-kube-api-access-6pskr" seLinuxMountContext=""
Oct 29 22:42:44.453120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453042    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368c881a-7e15-4fad-8acf-2253c547cc54" volumeName="kubernetes.io/empty-dir/368c881a-7e15-4fad-8acf-2253c547cc54-data-volume" seLinuxMountContext=""
Oct 29 22:42:44.453120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453056    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368c881a-7e15-4fad-8acf-2253c547cc54" volumeName="kubernetes.io/secret/368c881a-7e15-4fad-8acf-2253c547cc54-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 29 22:42:44.453120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453068    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="becd3290-1b8d-45ef-8381-b2aec149f522" volumeName="kubernetes.io/projected/becd3290-1b8d-45ef-8381-b2aec149f522-kube-api-access-vp9pw" seLinuxMountContext=""
Oct 29 22:42:44.453120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453079    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d9a88e-975d-4596-9c23-1144707d5391" volumeName="kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-tmp" seLinuxMountContext=""
Oct 29 22:42:44.453120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453091    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8ed141-dd58-45a2-841e-22c1d8eb19c1" volumeName="kubernetes.io/projected/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-kube-api-access-64s69" seLinuxMountContext=""
Oct 29 22:42:44.453120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453101    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0e7869c-ca3a-4a1c-abac-f3bfa1639076" volumeName="kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth" seLinuxMountContext=""
Oct 29 22:42:44.453120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453112    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13e0ca11-32d7-4d64-9388-7f50197dd515" volumeName="kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453126    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34812142-44fe-4a32-b03e-97fc0a461bd5" volumeName="kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453138    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368c881a-7e15-4fad-8acf-2253c547cc54" volumeName="kubernetes.io/projected/368c881a-7e15-4fad-8acf-2253c547cc54-kube-api-access-b4fm4" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453149    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5612c1b5-6517-4a68-a0a4-3a10ff070d31" volumeName="kubernetes.io/configmap/5612c1b5-6517-4a68-a0a4-3a10ff070d31-serviceca" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453161    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5caa8e9-a021-4107-8fb3-945c7b1179e6" volumeName="kubernetes.io/configmap/b5caa8e9-a021-4107-8fb3-945c7b1179e6-iptables-alerter-script" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453172    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be8bd6d3-88a1-482b-a514-d254705c5cbc" volumeName="kubernetes.io/projected/be8bd6d3-88a1-482b-a514-d254705c5cbc-kube-api-access-frmnh" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453195    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4b0b131-a728-4cee-aa9e-efa80695c400" volumeName="kubernetes.io/empty-dir/d4b0b131-a728-4cee-aa9e-efa80695c400-tmp-dir" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453209    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3a24a6b-c505-47ea-9400-f359710c3883" volumeName="kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-multus-daemon-config" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453221    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3a24a6b-c505-47ea-9400-f359710c3883" volumeName="kubernetes.io/projected/b3a24a6b-c505-47ea-9400-f359710c3883-kube-api-access-n5vct" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453237    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be8bd6d3-88a1-482b-a514-d254705c5cbc" volumeName="kubernetes.io/empty-dir/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-textfile" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453249    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4b0b131-a728-4cee-aa9e-efa80695c400" volumeName="kubernetes.io/projected/d4b0b131-a728-4cee-aa9e-efa80695c400-kube-api-access-qlzvq" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453260    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8ed141-dd58-45a2-841e-22c1d8eb19c1" volumeName="kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-env-overrides" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453271    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8ed141-dd58-45a2-841e-22c1d8eb19c1" volumeName="kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-config" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453283    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79884975-9bf8-48b4-89c1-95e4f33e8285" volumeName="kubernetes.io/configmap/79884975-9bf8-48b4-89c1-95e4f33e8285-config-volume" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453299    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5029cc-b18a-4547-9e9e-7a172b56e789" volumeName="kubernetes.io/projected/8d5029cc-b18a-4547-9e9e-7a172b56e789-kube-api-access-xdxw7" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453307    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3a24a6b-c505-47ea-9400-f359710c3883" volumeName="kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453315    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5caa8e9-a021-4107-8fb3-945c7b1179e6" volumeName="kubernetes.io/projected/b5caa8e9-a021-4107-8fb3-945c7b1179e6-kube-api-access-xd7fl" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453323    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be8bd6d3-88a1-482b-a514-d254705c5cbc" volumeName="kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453331    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ea7bf0-7270-447b-8aeb-a91164efe631" volumeName="kubernetes.io/secret/c8ea7bf0-7270-447b-8aeb-a91164efe631-agent-certs" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453339    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1014721-61ff-4d6e-b0f7-83b4f51649c5" volumeName="kubernetes.io/projected/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kube-api-access-9lrnj" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453356    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8ed141-dd58-45a2-841e-22c1d8eb19c1" volumeName="kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453367    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368c881a-7e15-4fad-8acf-2253c547cc54" volumeName="kubernetes.io/configmap/368c881a-7e15-4fad-8acf-2253c547cc54-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453376    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5029cc-b18a-4547-9e9e-7a172b56e789" volumeName="kubernetes.io/secret/8d5029cc-b18a-4547-9e9e-7a172b56e789-cert" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453384    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8ed141-dd58-45a2-841e-22c1d8eb19c1" volumeName="kubernetes.io/secret/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453391    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13e0ca11-32d7-4d64-9388-7f50197dd515" volumeName="kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453407    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13e0ca11-32d7-4d64-9388-7f50197dd515" volumeName="kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453417    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be8bd6d3-88a1-482b-a514-d254705c5cbc" volumeName="kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-tls" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453426    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79884975-9bf8-48b4-89c1-95e4f33e8285" volumeName="kubernetes.io/projected/79884975-9bf8-48b4-89c1-95e4f33e8285-kube-api-access-6c5tr" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453434    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be8bd6d3-88a1-482b-a514-d254705c5cbc" volumeName="kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453442    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ea7bf0-7270-447b-8aeb-a91164efe631" volumeName="kubernetes.io/configmap/c8ea7bf0-7270-447b-8aeb-a91164efe631-konnectivity-ca" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453449    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d9a88e-975d-4596-9c23-1144707d5391" volumeName="kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-etc-tuned" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453456    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d9a88e-975d-4596-9c23-1144707d5391" volumeName="kubernetes.io/projected/c9d9a88e-975d-4596-9c23-1144707d5391-kube-api-access-z5pbs" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453464    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5612c1b5-6517-4a68-a0a4-3a10ff070d31" volumeName="kubernetes.io/projected/5612c1b5-6517-4a68-a0a4-3a10ff070d31-kube-api-access-slmrs" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453471    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79884975-9bf8-48b4-89c1-95e4f33e8285" volumeName="kubernetes.io/empty-dir/79884975-9bf8-48b4-89c1-95e4f33e8285-tmp-dir" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453480    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79884975-9bf8-48b4-89c1-95e4f33e8285" volumeName="kubernetes.io/secret/79884975-9bf8-48b4-89c1-95e4f33e8285-metrics-tls" seLinuxMountContext=""
Oct 29 22:42:44.453469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453487    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be8bd6d3-88a1-482b-a514-d254705c5cbc" volumeName="kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-metrics-client-ca" seLinuxMountContext=""
Oct 29 22:42:44.454997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453495    4827 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="becd3290-1b8d-45ef-8381-b2aec149f522" volumeName="kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs" seLinuxMountContext=""
Oct 29 22:42:44.454997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453503    4827 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:42:44.454997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.453508    4827 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:42:44.474631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.474611    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:42:44.476084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.476063    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:42:44.478330 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.478287    4827 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:42:44.479901 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.479874    4827 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:42:44.479901 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.479903    4827 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:42:44.480042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.479928    4827 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:42:44.480042 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:44.479966    4827 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:42:44.495105 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.495081    4827 generic.go:356] "Generic (PLEG): container finished" podID="3b311dfc9e200f9ebd4f84d0618b84c9" containerID="604ee44d6201d386ab4ca8fa62615ed3b7235713754da1f2857eb55fbc40e09c" exitCode=0
Oct 29 22:42:44.495926 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.495906    4827 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 22:42:44.497353 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.497333    4827 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:44.497436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.497385    4827 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:44.509289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.509269    4827 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:44.513990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.513929    4827 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="f7a753d2e725376cf1090115b0a9f14ddc21f02325b9b3a98a0def990e7ac8b4" exitCode=0
Oct 29 22:42:44.513990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.513957    4827 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="820eb9f4bb81f331ec3d0c4142d2370bf808f10c346c84ae9b61ed49af69e238" exitCode=0
Oct 29 22:42:44.513990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.513966    4827 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="34ba4b1a193542cef525f3d64b95c91296ac60366b6c78659a4150e7c829457e" exitCode=0
Oct 29 22:42:44.513990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.513974    4827 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="1107f18e808c1716b47484cef81eb5b112f4ea63f79578b484b59a14daa32694" exitCode=0
Oct 29 22:42:44.513990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.513981    4827 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="b9256bb962691ed927dd09792bbc62e2840cf2e8132fa2451be1b56ae6512bf8" exitCode=0
Oct 29 22:42:44.513990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.513988    4827 generic.go:356] "Generic (PLEG): container finished" podID="13e0ca11-32d7-4d64-9388-7f50197dd515" containerID="8999d5772e6d2e6dc893ddaa2f8d4236c1d293ab726a742ccb9f0e887ef8ff03" exitCode=0
Oct 29 22:42:44.518313 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.518295    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:42:44.518663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.518635    4827 generic.go:356] "Generic (PLEG): container finished" podID="dc8ed141-dd58-45a2-841e-22c1d8eb19c1" containerID="c1176da34524362fe16ee02ba33776ca5fc4c74d374a424798859d84698981c0" exitCode=1
Oct 29 22:42:44.519797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.519759    4827 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:44.520984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.520960    4827 generic.go:356] "Generic (PLEG): container finished" podID="be8bd6d3-88a1-482b-a514-d254705c5cbc" containerID="ce4853e825378dbc905f1991e44b9be4c4bce003416234177b199fa8ed67f9b8" exitCode=0
Oct 29 22:42:44.533551 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.533525    4827 manager.go:324] Recovery completed
Oct 29 22:42:44.550153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550132    4827 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:42:44.550153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550147    4827 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:42:44.550271 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550163    4827 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:42:44.550300 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550290    4827 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 22:42:44.550328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550301    4827 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 22:42:44.550328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550318    4827 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 22:42:44.550328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550323    4827 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 22:42:44.550409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550329    4827 policy_none.go:50] "Start"
Oct 29 22:42:44.550409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550337    4827 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:42:44.550409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550348    4827 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:42:44.550485 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550479    4827 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 22:42:44.550522 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550486    4827 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 22:42:44.550522 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.550491    4827 policy_none.go:44] "Start"
Oct 29 22:42:44.554535 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.554511    4827 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:42:44.557524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.557514    4827 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:42:44.558048 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:44.557536    4827 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:42:44.558048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558049    4827 server.go:87] "Starting device plugin registration server"
Oct 29 22:42:44.558290 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558280    4827 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:42:44.558321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558291    4827 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:42:44.558438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558414    4827 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:42:44.558572 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558534    4827 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:42:44.558638 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558583    4827 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:42:44.558638 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558590    4827 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:42:44.558754 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.558625    4827 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:44.55857079Z","UUID":"4c267528-dab3-448d-a2a7-28c60088019a","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:42:44.560140 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.560124    4827 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:42:44.560231 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.560149    4827 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:42:44.561295 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:44.561272    4827 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:42:44.580070 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580035    4827 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"]
Oct 29 22:42:44.580498 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580469    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal" event={"ID":"3b311dfc9e200f9ebd4f84d0618b84c9","Type":"ContainerStarted","Data":"59c78118a8e9da706415c2a6ac40715eddd268444a264357ff224ff1992f2042"}
Oct 29 22:42:44.580580 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580505    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal" event={"ID":"3b311dfc9e200f9ebd4f84d0618b84c9","Type":"ContainerDied","Data":"604ee44d6201d386ab4ca8fa62615ed3b7235713754da1f2857eb55fbc40e09c"}
Oct 29 22:42:44.580580 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580528    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal" event={"ID":"3b311dfc9e200f9ebd4f84d0618b84c9","Type":"ContainerStarted","Data":"f6c60568b9dba9bf6b5f32736027fcab2c8cbd360f137f340959b29d5ab80119"}
Oct 29 22:42:44.580686 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580593    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal" event={"ID":"cac70c0770f909e7c0598a461e924492","Type":"ContainerStarted","Data":"db8a6451d122c0ca64124b40dad5ca270910cded34e85c9bf425b700202d68fa"}
Oct 29 22:42:44.580686 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580605    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal" event={"ID":"cac70c0770f909e7c0598a461e924492","Type":"ContainerStarted","Data":"91c73073488a7664c1481f3721edb987c9f6d35b3ab3b4170d551b05a7293e3e"}
Oct 29 22:42:44.580686 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580629    4827 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.580686 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.580646    4827 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.603736 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.603719    4827 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:44.603811 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:44.603760    4827 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.604413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.604402    4827 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:44.604447 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:44.604431    4827 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.659362 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.659339    4827 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:44.663149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.663130    4827 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:44.663248 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.663165    4827 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:44.663248 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.663179    4827 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-205.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:44.663248 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.663222    4827 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.696452 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.696428    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.696567 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.696459    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.696567 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.696488    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cac70c0770f909e7c0598a461e924492-config\") pod \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" (UID: \"cac70c0770f909e7c0598a461e924492\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.698030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.698014    4827 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.698081 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.698071    4827 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.797778 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.797683    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cac70c0770f909e7c0598a461e924492-config\") pod \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" (UID: \"cac70c0770f909e7c0598a461e924492\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.797778 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.797695    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cac70c0770f909e7c0598a461e924492-config\") pod \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" (UID: \"cac70c0770f909e7c0598a461e924492\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.797778 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.797754    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.797979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.797800    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.797979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.797847    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:44.797979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:44.797848    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b311dfc9e200f9ebd4f84d0618b84c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" (UID: \"3b311dfc9e200f9ebd4f84d0618b84c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:45.436092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.436058    4827 apiserver.go:52] "Watching apiserver"
Oct 29 22:42:45.485705 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.485676    4827 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:42:45.486226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.486201    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-mcxlq","openshift-multus/network-metrics-daemon-nfc6s","openshift-insights/insights-runtime-extractor-7xwpt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal","openshift-network-operator/iptables-alerter-4vngd","openshift-ovn-kubernetes/ovnkube-node-2sbt6","kube-system/global-pull-secret-syncer-nt7zn","openshift-dns/dns-default-vqfnh","openshift-image-registry/node-ca-76p4s","openshift-monitoring/node-exporter-zspc8","kube-system/konnectivity-agent-hc2bd","openshift-cluster-node-tuning-operator/tuned-867zt","openshift-multus/multus-additional-cni-plugins-42xsf","kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal","openshift-network-diagnostics/network-check-target-649bp","openshift-ingress-canary/ingress-canary-f67rh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t","openshift-dns/node-resolver-hg7lt"]
Oct 29 22:42:45.501027 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501003    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.501122 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501048    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.501122 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501068    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:42:45.501122 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501099    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:42:45.501122 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501114    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.501268 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501146    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6rt5\""
Oct 29 22:42:45.501268 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501183    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9xcm\""
Oct 29 22:42:45.501488 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.501467    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mhfn7\""
Oct 29 22:42:45.511979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.511948    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzwzj\""
Oct 29 22:42:45.511979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.511964    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:42:45.512092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.511988    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:42:45.512092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512042    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg4pb\""
Oct 29 22:42:45.512092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512064    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrmx7\""
Oct 29 22:42:45.512560 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512545    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:42:45.512614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512572    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:42:45.512614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512590    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pd9xt\""
Oct 29 22:42:45.512614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512605    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:42:45.512614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512641    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vd45d\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512681    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c4xfj\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512741    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512793    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wccb2\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512838    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g9cd8\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512899    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dvjrv\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512906    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512939    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512962    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l2jp\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512967    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512973    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512978    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512985    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.512990    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513000    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513046    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.513171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513135    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rfj6j\""
Oct 29 22:42:45.513733 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513197    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kr4qv\""
Oct 29 22:42:45.513733 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513418    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.513733 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513498    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:42:45.513733 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513505    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.513982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.513966    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:42:45.514030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514001    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgvrg\""
Oct 29 22:42:45.514030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514012    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.514030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514028    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:42:45.514351 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514257    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:42:45.514351 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514263    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.514669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514400    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.514669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514464    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.514669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514528    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.514669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514531    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:42:45.514669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514599    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.514669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514607    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:42:45.514669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514632    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:42:45.514936 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514738    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.514936 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.514748    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.523040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523028    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.523656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523638    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:42:45.523743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523669    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:42:45.523743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523682    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:42:45.523743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523695    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:42:45.523743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523710    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:42:45.523743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523719    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:42:45.523966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523808    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:42:45.523966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.523880    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:42:45.533912 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.533895    4827 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:45.534055 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.534043    4827 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:45.550297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.550279    4827 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:42:45.556323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.556309    4827 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:45.556367 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:45.556360    4827 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-205.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:45.556812 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.556800    4827 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:45.556843 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:42:45.556833    4827 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-205.ec2.internal"
Oct 29 22:42:45.601419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601398    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-systemd\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.601419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601420    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-netns\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.601574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601445    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b0b131-a728-4cee-aa9e-efa80695c400-tmp-dir\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:45.601574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601460    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-tls\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.601574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601479    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5caa8e9-a021-4107-8fb3-945c7b1179e6-iptables-alerter-script\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:45.601574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601560    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-run\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.601738 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601596    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-lib-modules\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.601738 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601625    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-tmp\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.601738 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601659    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzvq\" (UniqueName: \"kubernetes.io/projected/d4b0b131-a728-4cee-aa9e-efa80695c400-kube-api-access-qlzvq\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:45.601738 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601714    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-tmp\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.601941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601758    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.601941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601802    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-netns\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.601941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601798    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b0b131-a728-4cee-aa9e-efa80695c400-tmp-dir\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:45.601941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601827    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-wtmp\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.601941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601858    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-dbus\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.601941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601902    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-system-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.601941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601923    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5caa8e9-a021-4107-8fb3-945c7b1179e6-iptables-alerter-script\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601962    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-modprobe-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.601996    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-tls\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-kubernetes\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602086    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602121    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9pw\" (UniqueName: \"kubernetes.io/projected/becd3290-1b8d-45ef-8381-b2aec149f522-kube-api-access-vp9pw\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602148    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79884975-9bf8-48b4-89c1-95e4f33e8285-tmp-dir\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602172    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d5029cc-b18a-4547-9e9e-7a172b56e789-cert\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602196    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602219    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-etc-tuned\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.602272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602252    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79884975-9bf8-48b4-89c1-95e4f33e8285-tmp-dir\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.602659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602525    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d9a88e-975d-4596-9c23-1144707d5391-etc-tuned\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.602659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602623    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d5029cc-b18a-4547-9e9e-7a172b56e789-cert\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:45.602659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602638    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrnj\" (UniqueName: \"kubernetes.io/projected/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kube-api-access-9lrnj\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.602801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602689    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-log-socket\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.602801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602722    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-bin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.602801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602788    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-cnibin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.602931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.602854    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-cni-binary-copy\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.603048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603031    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.603153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603074    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-binary-copy\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.603153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603146    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-slash\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.603284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603250    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-env-overrides\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.603426 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603410    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-cni-binary-copy\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.603457 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603436    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-binary-copy\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.603457 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603437    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-kubelet-config\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.603524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603490    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-conf-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.603553 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603535    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pskr\" (UniqueName: \"kubernetes.io/projected/13e0ca11-32d7-4d64-9388-7f50197dd515-kube-api-access-6pskr\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.603595 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603577    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-config\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.603644 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603612    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-root\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.603694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603645    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-metrics-client-ca\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.603694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603678    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5caa8e9-a021-4107-8fb3-945c7b1179e6-host-slash\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:45.603796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603645    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-env-overrides\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.603796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603711    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-kubelet\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.603896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603826    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-socket-dir-parent\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.603896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603856    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-sys\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.603982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603919    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:45.603982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603941    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-metrics-client-ca\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.603982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603948    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-config\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.603982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.603959    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8ea7bf0-7270-447b-8aeb-a91164efe631-konnectivity-ca\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:45.604231 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604212    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8ea7bf0-7270-447b-8aeb-a91164efe631-konnectivity-ca\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:45.604299 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604266    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79884975-9bf8-48b4-89c1-95e4f33e8285-config-volume\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.604360 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604294    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxw7\" (UniqueName: \"kubernetes.io/projected/8d5029cc-b18a-4547-9e9e-7a172b56e789-kube-api-access-xdxw7\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:45.604420 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604396    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-multus\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.604467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604424    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-cnibin\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.604467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604456    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-systemd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.604594 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604484    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-etc-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.604594 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604515    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-ovn\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.604594 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604580    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-script-lib\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.604746 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604613    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.604746 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604639    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.604746 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604672    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.604746 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604700    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-system-cni-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.604746 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604730    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.605030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604759    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-multus-daemon-config\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.605030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604799    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-etc-kubernetes\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.605030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604827    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-hostroot\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.605030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604856    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-var-lib-kubelet\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.605030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.604994    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vct\" (UniqueName: \"kubernetes.io/projected/b3a24a6b-c505-47ea-9400-f359710c3883-kube-api-access-n5vct\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.605030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605026    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysconfig\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605055    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-device-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605087    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/368c881a-7e15-4fad-8acf-2253c547cc54-data-volume\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605116    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-systemd-units\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605142    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-multus-certs\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605173    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605206    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/368c881a-7e15-4fad-8acf-2253c547cc54-crio-socket\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605233    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-sys\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605264    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8ea7bf0-7270-447b-8aeb-a91164efe631-agent-certs\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605301    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pbs\" (UniqueName: \"kubernetes.io/projected/c9d9a88e-975d-4596-9c23-1144707d5391-kube-api-access-z5pbs\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605328    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605356    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-kubelet\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605385    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79884975-9bf8-48b4-89c1-95e4f33e8285-metrics-tls\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.605427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605417    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmrs\" (UniqueName: \"kubernetes.io/projected/5612c1b5-6517-4a68-a0a4-3a10ff070d31-kube-api-access-slmrs\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605448    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605474    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-k8s-cni-cncf-io\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605504    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4b0b131-a728-4cee-aa9e-efa80695c400-hosts-file\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605537    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605709    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fm4\" (UniqueName: \"kubernetes.io/projected/368c881a-7e15-4fad-8acf-2253c547cc54-kube-api-access-b4fm4\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605740    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmnh\" (UniqueName: \"kubernetes.io/projected/be8bd6d3-88a1-482b-a514-d254705c5cbc-kube-api-access-frmnh\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605788    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605818    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5612c1b5-6517-4a68-a0a4-3a10ff070d31-serviceca\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605852    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-host\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605882    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605914    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/368c881a-7e15-4fad-8acf-2253c547cc54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605940    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-node-log\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605969    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-netd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.605998    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-textfile\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606029    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5tr\" (UniqueName: \"kubernetes.io/projected/79884975-9bf8-48b4-89c1-95e4f33e8285-kube-api-access-6c5tr\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606060    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-conf\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606083    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-bin\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.606104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606113    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovn-node-metrics-cert\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606141    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5612c1b5-6517-4a68-a0a4-3a10ff070d31-host\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606170    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-var-lib-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606201    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606227    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606259    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64s69\" (UniqueName: \"kubernetes.io/projected/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-kube-api-access-64s69\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606290    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606321    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7fl\" (UniqueName: \"kubernetes.io/projected/b5caa8e9-a021-4107-8fb3-945c7b1179e6-kube-api-access-xd7fl\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606349    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/368c881a-7e15-4fad-8acf-2253c547cc54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606783    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8ea7bf0-7270-447b-8aeb-a91164efe631-agent-certs\") pod \"konnectivity-agent-hc2bd\" (UID: \"c8ea7bf0-7270-447b-8aeb-a91164efe631\") " pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606826    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/368c881a-7e15-4fad-8acf-2253c547cc54-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606904    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/368c881a-7e15-4fad-8acf-2253c547cc54-data-volume\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606960    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79884975-9bf8-48b4-89c1-95e4f33e8285-config-volume\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.607016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.606965    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34812142-44fe-4a32-b03e-97fc0a461bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607045    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607173    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5612c1b5-6517-4a68-a0a4-3a10ff070d31-serviceca\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607191    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607203    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607291    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79884975-9bf8-48b4-89c1-95e4f33e8285-metrics-tls\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607291    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/368c881a-7e15-4fad-8acf-2253c547cc54-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607332    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovn-node-metrics-cert\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.607561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607371    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-textfile\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.607972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607954    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3a24a6b-c505-47ea-9400-f359710c3883-multus-daemon-config\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.608042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.607976    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13e0ca11-32d7-4d64-9388-7f50197dd515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.608089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.608038    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-ovnkube-script-lib\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.608128 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.608083    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becd3290-1b8d-45ef-8381-b2aec149f522-metrics-certs\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:45.622922 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.622903    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9pw\" (UniqueName: \"kubernetes.io/projected/becd3290-1b8d-45ef-8381-b2aec149f522-kube-api-access-vp9pw\") pod \"network-metrics-daemon-nfc6s\" (UID: \"becd3290-1b8d-45ef-8381-b2aec149f522\") " pod="openshift-multus/network-metrics-daemon-nfc6s"
Oct 29 22:42:45.623054 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.623037    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrnj\" (UniqueName: \"kubernetes.io/projected/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kube-api-access-9lrnj\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.623129 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.623113    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzvq\" (UniqueName: \"kubernetes.io/projected/d4b0b131-a728-4cee-aa9e-efa80695c400-kube-api-access-qlzvq\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:45.623384 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.623368    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pskr\" (UniqueName: \"kubernetes.io/projected/13e0ca11-32d7-4d64-9388-7f50197dd515-kube-api-access-6pskr\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.630127 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630106    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vct\" (UniqueName: \"kubernetes.io/projected/b3a24a6b-c505-47ea-9400-f359710c3883-kube-api-access-n5vct\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.630127 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630131    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7fl\" (UniqueName: \"kubernetes.io/projected/b5caa8e9-a021-4107-8fb3-945c7b1179e6-kube-api-access-xd7fl\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630567    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmnh\" (UniqueName: \"kubernetes.io/projected/be8bd6d3-88a1-482b-a514-d254705c5cbc-kube-api-access-frmnh\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630600    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmrs\" (UniqueName: \"kubernetes.io/projected/5612c1b5-6517-4a68-a0a4-3a10ff070d31-kube-api-access-slmrs\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630602    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fm4\" (UniqueName: \"kubernetes.io/projected/368c881a-7e15-4fad-8acf-2253c547cc54-kube-api-access-b4fm4\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630734    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pbs\" (UniqueName: \"kubernetes.io/projected/c9d9a88e-975d-4596-9c23-1144707d5391-kube-api-access-z5pbs\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630753    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64s69\" (UniqueName: \"kubernetes.io/projected/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-kube-api-access-64s69\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630780    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxw7\" (UniqueName: \"kubernetes.io/projected/8d5029cc-b18a-4547-9e9e-7a172b56e789-kube-api-access-xdxw7\") pod \"ingress-canary-f67rh\" (UID: \"8d5029cc-b18a-4547-9e9e-7a172b56e789\") " pod="openshift-ingress-canary/ingress-canary-f67rh"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630791    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvth\" (UniqueName: \"kubernetes.io/projected/e0e7869c-ca3a-4a1c-abac-f3bfa1639076-kube-api-access-5qvth\") pod \"network-check-target-649bp\" (UID: \"e0e7869c-ca3a-4a1c-abac-f3bfa1639076\") " pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:45.630844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.630816    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5tr\" (UniqueName: \"kubernetes.io/projected/79884975-9bf8-48b4-89c1-95e4f33e8285-kube-api-access-6c5tr\") pod \"dns-default-vqfnh\" (UID: \"79884975-9bf8-48b4-89c1-95e4f33e8285\") " pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:45.706948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706864    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-var-lib-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.706948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706894    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.706948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706913    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.706948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706938    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-systemd\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706957    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-netns\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706974    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-run\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706983    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-var-lib-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707033    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707046    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-run\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707056    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-systemd\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706988    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707073    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-netns\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.706989    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-lib-modules\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707106    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707129    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-netns\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707152    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-wtmp\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707170    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-lib-modules\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707173    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-dbus\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707206    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-system-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707223    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-dbus\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.707279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707231    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-modprobe-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707318    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-modprobe-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707347    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-node-exporter-wtmp\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707360    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-run-netns\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707386    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-kubernetes\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707392    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-system-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707430    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-kubernetes\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707441    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707452    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707487    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707492    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707515    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-log-socket\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707536    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-bin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707558    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-cnibin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707581    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707585    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-d\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707603    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-slash\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707631    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-bin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707662    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-kubelet-config\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707682    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-cnibin\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707687    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-conf-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707712    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-root\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707730    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707737    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5caa8e9-a021-4107-8fb3-945c7b1179e6-host-slash\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707778    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-kubelet\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707786    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34812142-44fe-4a32-b03e-97fc0a461bd5-kubelet-config\") pod \"global-pull-secret-syncer-nt7zn\" (UID: \"34812142-44fe-4a32-b03e-97fc0a461bd5\") " pod="kube-system/global-pull-secret-syncer-nt7zn"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707802    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-socket-dir-parent\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707828    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-sys\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707857    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-multus\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707882    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-cnibin\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707884    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-kubelet\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707906    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-systemd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707917    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-conf-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707932    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-etc-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707956    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-root\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707956    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-ovn\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707990    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707993    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-ovn\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708011    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-system-cni-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708038    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-etc-kubernetes\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708046    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-socket-dir-parent\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708057    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-hostroot\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708079    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-var-lib-kubelet\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.708067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708094    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-sys\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707821    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-log-socket\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708111    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysconfig\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.707859    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-slash\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708134    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-device-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708156    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-systemd-units\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708162    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-cnibin\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708176    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-multus-certs\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708198    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708206    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-run-systemd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708221    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/368c881a-7e15-4fad-8acf-2253c547cc54-crio-socket\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708241    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-sys\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-etc-openvswitch\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708267    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708288    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-kubelet\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708291    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysconfig\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708313    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708329    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5caa8e9-a021-4107-8fb3-945c7b1179e6-host-slash\") pod \"iptables-alerter-4vngd\" (UID: \"b5caa8e9-a021-4107-8fb3-945c7b1179e6\") " pod="openshift-network-operator/iptables-alerter-4vngd"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708337    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-k8s-cni-cncf-io\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708360    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4b0b131-a728-4cee-aa9e-efa80695c400-hosts-file\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708375    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708384    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708412    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-host\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708415    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-system-cni-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708434    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-node-log\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708457    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-netd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708459    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-var-lib-kubelet\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708482    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-conf\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708491    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-etc-kubernetes\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708502    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-kubelet\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708510    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-bin\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708535    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5612c1b5-6517-4a68-a0a4-3a10ff070d31-host\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708549    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-var-lib-cni-multus\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708553    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be8bd6d3-88a1-482b-a514-d254705c5cbc-sys\") pod \"node-exporter-zspc8\" (UID: \"be8bd6d3-88a1-482b-a514-d254705c5cbc\") " pod="openshift-monitoring/node-exporter-zspc8"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708599    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5612c1b5-6517-4a68-a0a4-3a10ff070d31-host\") pod \"node-ca-76p4s\" (UID: \"5612c1b5-6517-4a68-a0a4-3a10ff070d31\") " pod="openshift-image-registry/node-ca-76p4s"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708600    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/368c881a-7e15-4fad-8acf-2253c547cc54-crio-socket\") pod \"insights-runtime-extractor-7xwpt\" (UID: \"368c881a-7e15-4fad-8acf-2253c547cc54\") " pod="openshift-insights/insights-runtime-extractor-7xwpt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708606    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e0ca11-32d7-4d64-9388-7f50197dd515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42xsf\" (UID: \"13e0ca11-32d7-4d64-9388-7f50197dd515\") " pod="openshift-multus/multus-additional-cni-plugins-42xsf"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708630    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-multus-certs\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708654    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-host-run-k8s-cni-cncf-io\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708663    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-multus-cni-dir\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708685    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-device-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708703    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-netd\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708706    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4b0b131-a728-4cee-aa9e-efa80695c400-hosts-file\") pod \"node-resolver-hg7lt\" (UID: \"d4b0b131-a728-4cee-aa9e-efa80695c400\") " pod="openshift-dns/node-resolver-hg7lt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708452    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3a24a6b-c505-47ea-9400-f359710c3883-hostroot\") pod \"multus-mcxlq\" (UID: \"b3a24a6b-c505-47ea-9400-f359710c3883\") " pod="openshift-multus/multus-mcxlq"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708725    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-systemd-units\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708759    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708792    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-host\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708820    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-node-log\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708853    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8ed141-dd58-45a2-841e-22c1d8eb19c1-host-cni-bin\") pod \"ovnkube-node-2sbt6\" (UID: \"dc8ed141-dd58-45a2-841e-22c1d8eb19c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708855    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d9a88e-975d-4596-9c23-1144707d5391-etc-sysctl-conf\") pod \"tuned-867zt\" (UID: \"c9d9a88e-975d-4596-9c23-1144707d5391\") " pod="openshift-cluster-node-tuning-operator/tuned-867zt"
Oct 29 22:42:45.709507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:45.708893    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1014721-61ff-4d6e-b0f7-83b4f51649c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv8t\" (UID: \"d1014721-61ff-4d6e-b0f7-83b4f51649c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv8t"
Oct 29 22:42:47.156112 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:47.156086    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:47.267986 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:47.267961    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:42:47.688375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:47.688347    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:47.690204 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:47.690181    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vqfnh"
Oct 29 22:42:47.722697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:47.722677    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hc2bd"
Oct 29 22:42:55.080621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:55.080594    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:42:55.081020 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:42:55.080876    4827 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:43:02.821324 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:43:02.821287    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:43:13.165022 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:43:13.164988    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sbt6"
Oct 29 22:43:17.270578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:43:17.270548    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-649bp"
Oct 29 22:43:32.821451 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:43:32.821367    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:44:02.820857 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:44:02.820821    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:44:32.821627 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:44:32.821588    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:44:34.889265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.889236    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"]
Oct 29 22:44:34.891162 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.891147    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:34.905686 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.905664    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:44:34.905827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.905696    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 22:44:34.905827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.905754    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:44:34.916982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.916961    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:44:34.917064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.916985    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:44:34.917064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.917033    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:44:34.917150 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.917113    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dpk7p\""
Oct 29 22:44:34.920620 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.920604    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"]
Oct 29 22:44:34.950903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.950871    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6rqw\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-kube-api-access-m6rqw\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:34.951004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.950905    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:34.951004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.950926    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07db380-14ed-49cb-b871-ef22cdee2cf8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:34.951004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.950991    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:34.951125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.951078    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:34.951125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.951110    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:34.951199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:34.951128    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052019 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.051984    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052027    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052045    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052072    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6rqw\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-kube-api-access-m6rqw\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052091    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052108    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07db380-14ed-49cb-b871-ef22cdee2cf8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052122    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.052501 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052473    4827 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:44:35.052647 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.052623    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.054552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.054511    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07db380-14ed-49cb-b871-ef22cdee2cf8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.054725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.054711    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.054844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.054824    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.054883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.054834    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.075146 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.075124    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6rqw\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-kube-api-access-m6rqw\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.075235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.075157    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lb5s7\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.201567 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.201489    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:35.508717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.508599    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"]
Oct 29 22:44:35.512761 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:44:35.512736    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07db380_14ed_49cb_b871_ef22cdee2cf8.slice/crio-fb7560696e3707af293971b6550194d2f1c1dd62a5b0d685d9c1944262b41c13 WatchSource:0}: Error finding container fb7560696e3707af293971b6550194d2f1c1dd62a5b0d685d9c1944262b41c13: Status 404 returned error can't find the container with id fb7560696e3707af293971b6550194d2f1c1dd62a5b0d685d9c1944262b41c13
Oct 29 22:44:35.514654 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.514632    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:44:35.854430 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:35.854349    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7" event={"ID":"b07db380-14ed-49cb-b871-ef22cdee2cf8","Type":"ContainerStarted","Data":"fb7560696e3707af293971b6550194d2f1c1dd62a5b0d685d9c1944262b41c13"}
Oct 29 22:44:38.062363 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:38.062330    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:44:38.062647 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:38.062390    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:44:38.865669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:38.865632    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7" event={"ID":"b07db380-14ed-49cb-b871-ef22cdee2cf8","Type":"ContainerStarted","Data":"8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b"}
Oct 29 22:44:38.865861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:38.865781    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:44:38.934326 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:38.934279    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7" podStartSLOduration=2.387529241 podStartE2EDuration="4.934266352s" totalImagesPullingTime="2.546737111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:38.922629169 +0000 UTC m=+114.545540191" watchObservedRunningTime="2026-10-29 22:44:38.934266352 +0000 UTC m=+114.557177372"
Oct 29 22:44:39.870491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:44:39.870462    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:45:02.821173 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:45:02.821092    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:45:15.729151 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.729119    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"]
Oct 29 22:45:15.736759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.736739    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.749321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.749299    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"]
Oct 29 22:45:15.810859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.810830    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.810859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.810865    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlbd\" (UniqueName: \"kubernetes.io/projected/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-kube-api-access-2zlbd\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.811086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.810886    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.811086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.810903    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.811086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.811053    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.811199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.811091    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.814715 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.814695    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 22:45:15.814808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.814723    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 29 22:45:15.814808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.814753    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 29 22:45:15.814808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.814791    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 22:45:15.814930 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.814852    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 29 22:45:15.814965 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.814930    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6q6mp\""
Oct 29 22:45:15.911976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.911952    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.912117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.911982    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlbd\" (UniqueName: \"kubernetes.io/projected/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-kube-api-access-2zlbd\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.912117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.912008    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.912117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.912024    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.912272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.912215    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.912318 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.912274    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.912507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.912484    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.914550 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.914522    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.914642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.914612    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.914697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.914676    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.914741 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.914728    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:15.934360 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:15.934339    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlbd\" (UniqueName: \"kubernetes.io/projected/42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4-kube-api-access-2zlbd\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-cgmw8\" (UID: \"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:16.051680 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:16.051597    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:16.401601 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:16.401475    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"]
Oct 29 22:45:16.404411 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:45:16.404385    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c9d8ba_36a2_4f2b_80f6_3f6295fa86b4.slice/crio-b6a48b6e60d5f80293d18d94154c5013051e046cc57703127a8032e5dd8c03cb WatchSource:0}: Error finding container b6a48b6e60d5f80293d18d94154c5013051e046cc57703127a8032e5dd8c03cb: Status 404 returned error can't find the container with id b6a48b6e60d5f80293d18d94154c5013051e046cc57703127a8032e5dd8c03cb
Oct 29 22:45:16.978503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:16.978464    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8" event={"ID":"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4","Type":"ContainerStarted","Data":"b6a48b6e60d5f80293d18d94154c5013051e046cc57703127a8032e5dd8c03cb"}
Oct 29 22:45:19.989951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:19.989862    4827 generic.go:356] "Generic (PLEG): container finished" podID="42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4" containerID="81eb918dd6c30a8162dfff0779bda105cdb4facd3857e1b267a2f75f6fc3306b" exitCode=0
Oct 29 22:45:19.989951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:19.989913    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8" event={"ID":"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4","Type":"ContainerDied","Data":"81eb918dd6c30a8162dfff0779bda105cdb4facd3857e1b267a2f75f6fc3306b"}
Oct 29 22:45:20.996265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:20.996228    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8" event={"ID":"42c9d8ba-36a2-4f2b-80f6-3f6295fa86b4","Type":"ContainerStarted","Data":"0c48bf74d17b87d834500857d1180203266c52fadad31619af3d37f71cb9880e"}
Oct 29 22:45:20.996715 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:20.996361    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:21.088513 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:21.088429    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8" podStartSLOduration=2.558422383 podStartE2EDuration="6.088416646s" totalImagesPullingTime="3.077766026s" totalInitContainerRuntime="452.228237ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:21.078304915 +0000 UTC m=+156.701215948" watchObservedRunningTime="2026-10-29 22:45:21.088416646 +0000 UTC m=+156.711327667"
Oct 29 22:45:32.001785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:32.001733    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-cgmw8"
Oct 29 22:45:32.823531 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:45:32.823491    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:45:46.106702 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.106672    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"]
Oct 29 22:45:46.112548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.112528    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.130359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.130342    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5rrn8\""
Oct 29 22:45:46.142249 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.142228    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 22:45:46.142371 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.142315    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 22:45:46.142958 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.142936    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"]
Oct 29 22:45:46.227037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.227009    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636f1206-971e-4ef8-be31-8f2e6564589c-tls-certs\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.227180 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.227054    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/636f1206-971e-4ef8-be31-8f2e6564589c-tmp\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.227180 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.227073    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvjr\" (UniqueName: \"kubernetes.io/projected/636f1206-971e-4ef8-be31-8f2e6564589c-kube-api-access-kzvjr\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.325721 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.325687    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"]
Oct 29 22:45:46.327963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.327941    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.328149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.328122    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/636f1206-971e-4ef8-be31-8f2e6564589c-tmp\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.328241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.328169    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvjr\" (UniqueName: \"kubernetes.io/projected/636f1206-971e-4ef8-be31-8f2e6564589c-kube-api-access-kzvjr\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.328241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.328221    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636f1206-971e-4ef8-be31-8f2e6564589c-tls-certs\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.330620 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.330592    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/636f1206-971e-4ef8-be31-8f2e6564589c-tmp\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.331922 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.331901    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636f1206-971e-4ef8-be31-8f2e6564589c-tls-certs\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.348013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.347985    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zbgkv\""
Oct 29 22:45:46.359080 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.359026    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvjr\" (UniqueName: \"kubernetes.io/projected/636f1206-971e-4ef8-be31-8f2e6564589c-kube-api-access-kzvjr\") pod \"kube-auth-proxy-5fbc7f7f7f-k4vrd\" (UID: \"636f1206-971e-4ef8-be31-8f2e6564589c\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.362740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.362716    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"]
Oct 29 22:45:46.423095 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.423060    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"
Oct 29 22:45:46.429164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429141    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429177    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d6d562-150a-4021-bf8b-153c01a7038d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429218    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429280    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429412 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429366    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429412 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429382    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429412 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429398    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429555 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429419    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.429555 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.429478    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpz7\" (UniqueName: \"kubernetes.io/projected/b2d6d562-150a-4021-bf8b-153c01a7038d-kube-api-access-nzpz7\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530236 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530202    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530236 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530238    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530259    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530287    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530387    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpz7\" (UniqueName: \"kubernetes.io/projected/b2d6d562-150a-4021-bf8b-153c01a7038d-kube-api-access-nzpz7\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530454    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530497    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d6d562-150a-4021-bf8b-153c01a7038d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530549    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530575    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530660    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530715    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.530972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.530949    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.531054 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.531029    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.531487 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.531464    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d6d562-150a-4021-bf8b-153c01a7038d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.532715 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.532693    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.533513 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.533491    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.554376 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.554352    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d6d562-150a-4021-bf8b-153c01a7038d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.554495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.554479    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpz7\" (UniqueName: \"kubernetes.io/projected/b2d6d562-150a-4021-bf8b-153c01a7038d-kube-api-access-nzpz7\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d52czbg\" (UID: \"b2d6d562-150a-4021-bf8b-153c01a7038d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.646707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.646679    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:46.715293 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.715019    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd"]
Oct 29 22:45:46.720195 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:45:46.720167    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636f1206_971e_4ef8_be31_8f2e6564589c.slice/crio-a98875c8952d40c6acda4e18a131b959e7f4fff9b6558a31053555ebbefdcae1 WatchSource:0}: Error finding container a98875c8952d40c6acda4e18a131b959e7f4fff9b6558a31053555ebbefdcae1: Status 404 returned error can't find the container with id a98875c8952d40c6acda4e18a131b959e7f4fff9b6558a31053555ebbefdcae1
Oct 29 22:45:46.955110 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:46.955034    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"]
Oct 29 22:45:46.957968 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:45:46.957946    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d6d562_150a_4021_bf8b_153c01a7038d.slice/crio-131c9719ee46c1bd5b5310229a632e1be7884379f662c30b865823134f7b0e6a WatchSource:0}: Error finding container 131c9719ee46c1bd5b5310229a632e1be7884379f662c30b865823134f7b0e6a: Status 404 returned error can't find the container with id 131c9719ee46c1bd5b5310229a632e1be7884379f662c30b865823134f7b0e6a
Oct 29 22:45:47.075220 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.075188    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd" event={"ID":"636f1206-971e-4ef8-be31-8f2e6564589c","Type":"ContainerStarted","Data":"a98875c8952d40c6acda4e18a131b959e7f4fff9b6558a31053555ebbefdcae1"}
Oct 29 22:45:47.076326 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.076302    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg" event={"ID":"b2d6d562-150a-4021-bf8b-153c01a7038d","Type":"ContainerStarted","Data":"131c9719ee46c1bd5b5310229a632e1be7884379f662c30b865823134f7b0e6a"}
Oct 29 22:45:47.547209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.547175    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"]
Oct 29 22:45:47.549550 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.549517    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"
Oct 29 22:45:47.564270 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.564230    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tg7tx\""
Oct 29 22:45:47.574897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.574871    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 22:45:47.574897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.574879    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 22:45:47.575985 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.575960    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"]
Oct 29 22:45:47.639936 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.639901    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hk6d\" (UniqueName: \"kubernetes.io/projected/9037b762-9013-436a-8abc-fc94d30b6f29-kube-api-access-8hk6d\") pod \"authorino-operator-7b7bb4bc77-cdfvp\" (UID: \"9037b762-9013-436a-8abc-fc94d30b6f29\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"
Oct 29 22:45:47.741295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.741265    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hk6d\" (UniqueName: \"kubernetes.io/projected/9037b762-9013-436a-8abc-fc94d30b6f29-kube-api-access-8hk6d\") pod \"authorino-operator-7b7bb4bc77-cdfvp\" (UID: \"9037b762-9013-436a-8abc-fc94d30b6f29\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"
Oct 29 22:45:47.767403 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.767375    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hk6d\" (UniqueName: \"kubernetes.io/projected/9037b762-9013-436a-8abc-fc94d30b6f29-kube-api-access-8hk6d\") pod \"authorino-operator-7b7bb4bc77-cdfvp\" (UID: \"9037b762-9013-436a-8abc-fc94d30b6f29\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"
Oct 29 22:45:47.866712 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:47.866628    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"
Oct 29 22:45:48.181424 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:48.181388    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"]
Oct 29 22:45:49.087707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:49.087564    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp" event={"ID":"9037b762-9013-436a-8abc-fc94d30b6f29","Type":"ContainerStarted","Data":"6d71878dbaad978d16fe4441c61a00c8c8aed5aa35bc3b5247999b717572a4e8"}
Oct 29 22:45:50.094064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:50.094025    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd" event={"ID":"636f1206-971e-4ef8-be31-8f2e6564589c","Type":"ContainerStarted","Data":"ee711ee03fca16a9671180ba852581b6329c7c0b28c1ec7c02de3e091a88c4fd"}
Oct 29 22:45:50.136367 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:50.136191    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-k4vrd" podStartSLOduration=1.403233173 podStartE2EDuration="4.136173667s" totalImagesPullingTime="2.732940494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:50.135335124 +0000 UTC m=+185.758246154" watchObservedRunningTime="2026-10-29 22:45:50.136173667 +0000 UTC m=+185.759084717"
Oct 29 22:45:50.387752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:50.387722    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:45:50.387903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:50.387821    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:45:50.387903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:50.387873    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:45:51.098784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:51.098731    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg" event={"ID":"b2d6d562-150a-4021-bf8b-153c01a7038d","Type":"ContainerStarted","Data":"93129b57796e6c8f8890a509003254ec3427e178a1bbd75f969098cc7c017a1d"}
Oct 29 22:45:51.100377 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:51.100350    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp" event={"ID":"9037b762-9013-436a-8abc-fc94d30b6f29","Type":"ContainerStarted","Data":"f26e6003cc33262302433711e578cce6aeb46217fb753f31eada76c5e7af35ea"}
Oct 29 22:45:51.175478 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:51.175431    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg" podStartSLOduration=1.748444903 podStartE2EDuration="5.175418481s" totalImagesPullingTime="3.426973578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:51.164144764 +0000 UTC m=+186.787055784" watchObservedRunningTime="2026-10-29 22:45:51.175418481 +0000 UTC m=+186.798329498"
Oct 29 22:45:51.222384 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:51.222306    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp" podStartSLOduration=1.478743153 podStartE2EDuration="4.222293352s" totalImagesPullingTime="2.743550199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:51.22204975 +0000 UTC m=+186.844960770" watchObservedRunningTime="2026-10-29 22:45:51.222293352 +0000 UTC m=+186.845204371"
Oct 29 22:45:51.647640 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:51.647599    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:51.652270 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:51.652246    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:52.103610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:52.103524    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:52.104096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:52.103616    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"
Oct 29 22:45:52.104735 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:52.104715    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d52czbg"
Oct 29 22:45:54.725200 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.725166    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"]
Oct 29 22:45:54.728416 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.728396    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.745949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.745927    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 22:45:54.757036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.757016    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 22:45:54.757350 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.757329    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-6f755\""
Oct 29 22:45:54.782100 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.782076    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 29 22:45:54.782427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.782411    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"]
Oct 29 22:45:54.796852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.796818    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f894835-c3c1-4d46-a6c6-ca6d686676fe-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.796984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.796909    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f894835-c3c1-4d46-a6c6-ca6d686676fe-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.796984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.796952    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f894835-c3c1-4d46-a6c6-ca6d686676fe-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.797065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.797018    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jgx\" (UniqueName: \"kubernetes.io/projected/3f894835-c3c1-4d46-a6c6-ca6d686676fe-kube-api-access-t8jgx\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.897618 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.897592    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f894835-c3c1-4d46-a6c6-ca6d686676fe-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.897795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.897627    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f894835-c3c1-4d46-a6c6-ca6d686676fe-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.897795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.897649    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f894835-c3c1-4d46-a6c6-ca6d686676fe-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.897795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.897683    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jgx\" (UniqueName: \"kubernetes.io/projected/3f894835-c3c1-4d46-a6c6-ca6d686676fe-kube-api-access-t8jgx\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.897994 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.897971    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f894835-c3c1-4d46-a6c6-ca6d686676fe-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.898074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.898052    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f894835-c3c1-4d46-a6c6-ca6d686676fe-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.898509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.898491    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f894835-c3c1-4d46-a6c6-ca6d686676fe-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:54.920216 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:54.920192    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jgx\" (UniqueName: \"kubernetes.io/projected/3f894835-c3c1-4d46-a6c6-ca6d686676fe-kube-api-access-t8jgx\") pod \"kserve-module-controller-manager-7566cf5fc8-65lbj\" (UID: \"3f894835-c3c1-4d46-a6c6-ca6d686676fe\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:55.040333 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:55.040256    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:55.345211 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:55.345131    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"]
Oct 29 22:45:55.348343 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:45:55.348319    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f894835_c3c1_4d46_a6c6_ca6d686676fe.slice/crio-648bf54779e0827f4918ceabcd665d29505b9b6994f00989784ce4eb06202f77 WatchSource:0}: Error finding container 648bf54779e0827f4918ceabcd665d29505b9b6994f00989784ce4eb06202f77: Status 404 returned error can't find the container with id 648bf54779e0827f4918ceabcd665d29505b9b6994f00989784ce4eb06202f77
Oct 29 22:45:56.119672 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:56.119637    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj" event={"ID":"3f894835-c3c1-4d46-a6c6-ca6d686676fe","Type":"ContainerStarted","Data":"648bf54779e0827f4918ceabcd665d29505b9b6994f00989784ce4eb06202f77"}
Oct 29 22:45:57.124218 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:57.124178    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj" event={"ID":"3f894835-c3c1-4d46-a6c6-ca6d686676fe","Type":"ContainerStarted","Data":"7b0ae2535f37285648e24f59fab85ed4635eca528232bbfb901e9821f6a7914a"}
Oct 29 22:45:57.124690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:57.124301    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:45:57.177609 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:45:57.177552    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj" podStartSLOduration=1.846490816 podStartE2EDuration="3.17753993s" totalImagesPullingTime="1.331049114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:57.177165587 +0000 UTC m=+192.800076607" watchObservedRunningTime="2026-10-29 22:45:57.17753993 +0000 UTC m=+192.800450999"
Oct 29 22:46:02.821740 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:46:02.821699    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:46:03.109462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:03.109389    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cdfvp"
Oct 29 22:46:18.128744 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:18.128711    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-65lbj"
Oct 29 22:46:32.821692 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:46:32.821610    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:46:34.432680 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.432648    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zgj2q"]
Oct 29 22:46:34.434840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.434810    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zgj2q"
Oct 29 22:46:34.448991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.448971    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ppj7r\""
Oct 29 22:46:34.452188 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.452168    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zgj2q"]
Oct 29 22:46:34.495466 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.495434    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pf4f\" (UniqueName: \"kubernetes.io/projected/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c-kube-api-access-7pf4f\") pod \"authorino-57f7f8659f-zgj2q\" (UID: \"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c\") " pod="kuadrant-system/authorino-57f7f8659f-zgj2q"
Oct 29 22:46:34.596603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.596567    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pf4f\" (UniqueName: \"kubernetes.io/projected/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c-kube-api-access-7pf4f\") pod \"authorino-57f7f8659f-zgj2q\" (UID: \"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c\") " pod="kuadrant-system/authorino-57f7f8659f-zgj2q"
Oct 29 22:46:34.619624 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.619591    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pf4f\" (UniqueName: \"kubernetes.io/projected/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c-kube-api-access-7pf4f\") pod \"authorino-57f7f8659f-zgj2q\" (UID: \"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c\") " pod="kuadrant-system/authorino-57f7f8659f-zgj2q"
Oct 29 22:46:34.758748 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:34.758678    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zgj2q"
Oct 29 22:46:35.052300 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:35.052228    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zgj2q"]
Oct 29 22:46:35.055440 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:46:35.055414    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4967eb5_eaa7_44db_8e7f_67e67cac4f1c.slice/crio-aed8c5fe3a174e5acd5e3ca2c95e249fc0bd95d0dab9af9f5e2e26e8b2887078 WatchSource:0}: Error finding container aed8c5fe3a174e5acd5e3ca2c95e249fc0bd95d0dab9af9f5e2e26e8b2887078: Status 404 returned error can't find the container with id aed8c5fe3a174e5acd5e3ca2c95e249fc0bd95d0dab9af9f5e2e26e8b2887078
Oct 29 22:46:35.243415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:35.243382    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zgj2q" event={"ID":"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c","Type":"ContainerStarted","Data":"aed8c5fe3a174e5acd5e3ca2c95e249fc0bd95d0dab9af9f5e2e26e8b2887078"}
Oct 29 22:46:37.683321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:37.683287    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zgj2q"]
Oct 29 22:46:38.255367 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:38.255331    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zgj2q" event={"ID":"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c","Type":"ContainerStarted","Data":"aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad"}
Oct 29 22:46:38.293621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:38.293563    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-zgj2q" podStartSLOduration=2.02162185 podStartE2EDuration="4.293548532s" totalImagesPullingTime="2.271926682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:38.292833539 +0000 UTC m=+233.915744560" watchObservedRunningTime="2026-10-29 22:46:38.293548532 +0000 UTC m=+233.916459594"
Oct 29 22:46:39.258994 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:39.258952    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-zgj2q" podUID="f4967eb5-eaa7-44db-8e7f-67e67cac4f1c" containerName="authorino" containerID="cri-o://aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad" gracePeriod=30
Oct 29 22:46:39.495224 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:39.495202    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zgj2q"
Oct 29 22:46:39.537492 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:39.537425    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c-kube-api-access-7pf4f\" (UniqueName: \"kubernetes.io/projected/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c-kube-api-access-7pf4f\") pod \"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c\" (UID: \"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c\") "
Oct 29 22:46:39.539614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:39.539587    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c-kube-api-access-7pf4f" pod "f4967eb5-eaa7-44db-8e7f-67e67cac4f1c" (UID: "f4967eb5-eaa7-44db-8e7f-67e67cac4f1c"). InnerVolumeSpecName "kube-api-access-7pf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:39.638468 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:39.638434    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pf4f\" (UniqueName: \"kubernetes.io/projected/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c-kube-api-access-7pf4f\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:40.263274 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.263240    4827 generic.go:356] "Generic (PLEG): container finished" podID="f4967eb5-eaa7-44db-8e7f-67e67cac4f1c" containerID="aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad" exitCode=0
Oct 29 22:46:40.263732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.263288    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zgj2q" event={"ID":"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c","Type":"ContainerDied","Data":"aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad"}
Oct 29 22:46:40.263732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.263293    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zgj2q"
Oct 29 22:46:40.263732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.263325    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zgj2q" event={"ID":"f4967eb5-eaa7-44db-8e7f-67e67cac4f1c","Type":"ContainerDied","Data":"aed8c5fe3a174e5acd5e3ca2c95e249fc0bd95d0dab9af9f5e2e26e8b2887078"}
Oct 29 22:46:40.263732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.263343    4827 scope.go:122] "RemoveContainer" containerID="aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad"
Oct 29 22:46:40.271903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.271885    4827 scope.go:122] "RemoveContainer" containerID="aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad"
Oct 29 22:46:40.272249 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:46:40.272217    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad\": container with ID starting with aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad not found: ID does not exist" containerID="aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad"
Oct 29 22:46:40.272345 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.272258    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad"} err="failed to get container status \"aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad\": rpc error: code = NotFound desc = could not find container \"aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad\": container with ID starting with aaa1b7f51b59beaf8130ec1b3ff2e48ccf4bea9503750e086fafa01caeaac6ad not found: ID does not exist"
Oct 29 22:46:40.310377 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.310351    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zgj2q"]
Oct 29 22:46:40.316466 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.316445    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zgj2q"]
Oct 29 22:46:40.485381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:40.485349    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4967eb5-eaa7-44db-8e7f-67e67cac4f1c" path="/var/lib/kubelet/pods/f4967eb5-eaa7-44db-8e7f-67e67cac4f1c/volumes"
Oct 29 22:46:49.010794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.010748    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"]
Oct 29 22:46:49.011291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.011009    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f4967eb5-eaa7-44db-8e7f-67e67cac4f1c" containerName="authorino"
Oct 29 22:46:49.011291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.011019    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4967eb5-eaa7-44db-8e7f-67e67cac4f1c" containerName="authorino"
Oct 29 22:46:49.011291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.011073    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4967eb5-eaa7-44db-8e7f-67e67cac4f1c" containerName="authorino"
Oct 29 22:46:49.017301 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.017280    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.044166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.044144    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"]
Oct 29 22:46:49.108756 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.108717    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.108952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.108788    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eca953-9fa9-4c5b-b0ac-d88a23279013-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.108952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.108838    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.108952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.108869    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.108952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.108896    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.108952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.108918    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.108952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.108952    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznp9\" (UniqueName: \"kubernetes.io/projected/70eca953-9fa9-4c5b-b0ac-d88a23279013-kube-api-access-qznp9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.209365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.209325    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.209365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.209365    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eca953-9fa9-4c5b-b0ac-d88a23279013-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.209694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.209391    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.209694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.209513    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.209694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.209556    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.209694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.209577    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.209694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.209609    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznp9\" (UniqueName: \"kubernetes.io/projected/70eca953-9fa9-4c5b-b0ac-d88a23279013-kube-api-access-qznp9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.210225 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.210198    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.211849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.211827    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eca953-9fa9-4c5b-b0ac-d88a23279013-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.212020 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.211999    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.212346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.212320    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.212427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.212353    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.231919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.231896    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznp9\" (UniqueName: \"kubernetes.io/projected/70eca953-9fa9-4c5b-b0ac-d88a23279013-kube-api-access-qznp9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.232016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.231940    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70eca953-9fa9-4c5b-b0ac-d88a23279013-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wwjc\" (UID: \"70eca953-9fa9-4c5b-b0ac-d88a23279013\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.328190 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.328117    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:49.660919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.660882    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"]
Oct 29 22:46:49.665404 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:46:49.665374    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eca953_9fa9_4c5b_b0ac_d88a23279013.slice/crio-5280a5d1ef3cfe88c41bedaf44b74914ae8f7e84e7538ce4ddf2e349a3d7c8dc WatchSource:0}: Error finding container 5280a5d1ef3cfe88c41bedaf44b74914ae8f7e84e7538ce4ddf2e349a3d7c8dc: Status 404 returned error can't find the container with id 5280a5d1ef3cfe88c41bedaf44b74914ae8f7e84e7538ce4ddf2e349a3d7c8dc
Oct 29 22:46:49.667548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.667514    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:46:49.667607 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:49.667584    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:46:50.299071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:50.299037    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc" event={"ID":"70eca953-9fa9-4c5b-b0ac-d88a23279013","Type":"ContainerStarted","Data":"e7ae47a793b96c30a98ab4dd93ca1e3082f4c37c1d11131da2b1213bb466620c"}
Oct 29 22:46:50.299512 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:50.299087    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc" event={"ID":"70eca953-9fa9-4c5b-b0ac-d88a23279013","Type":"ContainerStarted","Data":"5280a5d1ef3cfe88c41bedaf44b74914ae8f7e84e7538ce4ddf2e349a3d7c8dc"}
Oct 29 22:46:50.299512 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:50.299180    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:50.300628 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:50.300601    4827 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7wwjc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:46:50.300750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:50.300653    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc" podUID="70eca953-9fa9-4c5b-b0ac-d88a23279013" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:46:50.347536 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:50.347487    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc" podStartSLOduration=2.347474565 podStartE2EDuration="2.347474565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:50.346450299 +0000 UTC m=+245.969361318" watchObservedRunningTime="2026-10-29 22:46:50.347474565 +0000 UTC m=+245.970385582"
Oct 29 22:46:51.302563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.302532    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wwjc"
Oct 29 22:46:51.392218 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.392190    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"]
Oct 29 22:46:51.392434 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.392414    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7" podUID="b07db380-14ed-49cb-b871-ef22cdee2cf8" containerName="discovery" containerID="cri-o://8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b" gracePeriod=30
Oct 29 22:46:51.628723 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.628701    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:46:51.733891 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.733849    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-token\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-token\") pod \"b07db380-14ed-49cb-b871-ef22cdee2cf8\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") "
Oct 29 22:46:51.734074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.733905    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07db380-14ed-49cb-b871-ef22cdee2cf8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07db380-14ed-49cb-b871-ef22cdee2cf8-local-certs\") pod \"b07db380-14ed-49cb-b871-ef22cdee2cf8\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") "
Oct 29 22:46:51.734074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.733937    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-kube-api-access-m6rqw\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-kube-api-access-m6rqw\") pod \"b07db380-14ed-49cb-b871-ef22cdee2cf8\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") "
Oct 29 22:46:51.734074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.733971    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-kubeconfig\") pod \"b07db380-14ed-49cb-b871-ef22cdee2cf8\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") "
Oct 29 22:46:51.734074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.734012    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-ca-configmap\") pod \"b07db380-14ed-49cb-b871-ef22cdee2cf8\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") "
Oct 29 22:46:51.734289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.734205    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-cacerts\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-cacerts\") pod \"b07db380-14ed-49cb-b871-ef22cdee2cf8\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") "
Oct 29 22:46:51.734349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.734290    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-dns-cert\") pod \"b07db380-14ed-49cb-b871-ef22cdee2cf8\" (UID: \"b07db380-14ed-49cb-b871-ef22cdee2cf8\") "
Oct 29 22:46:51.734401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.734378    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-ca-configmap" pod "b07db380-14ed-49cb-b871-ef22cdee2cf8" (UID: "b07db380-14ed-49cb-b871-ef22cdee2cf8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:51.734568 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.734551    4827 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-ca-configmap\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.736885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.736829    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-kubeconfig" pod "b07db380-14ed-49cb-b871-ef22cdee2cf8" (UID: "b07db380-14ed-49cb-b871-ef22cdee2cf8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:51.736885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.736848    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-cacerts" pod "b07db380-14ed-49cb-b871-ef22cdee2cf8" (UID: "b07db380-14ed-49cb-b871-ef22cdee2cf8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:51.737043 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.736877    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-token" pod "b07db380-14ed-49cb-b871-ef22cdee2cf8" (UID: "b07db380-14ed-49cb-b871-ef22cdee2cf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:51.737043 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.736890    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07db380-14ed-49cb-b871-ef22cdee2cf8-local-certs" pod "b07db380-14ed-49cb-b871-ef22cdee2cf8" (UID: "b07db380-14ed-49cb-b871-ef22cdee2cf8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:51.737043 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.736899    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-dns-cert" pod "b07db380-14ed-49cb-b871-ef22cdee2cf8" (UID: "b07db380-14ed-49cb-b871-ef22cdee2cf8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:51.737043 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.736924    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-kube-api-access-m6rqw" pod "b07db380-14ed-49cb-b871-ef22cdee2cf8" (UID: "b07db380-14ed-49cb-b871-ef22cdee2cf8"). InnerVolumeSpecName "kube-api-access-m6rqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:51.835047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.834975    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.835047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.835011    4827 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07db380-14ed-49cb-b871-ef22cdee2cf8-local-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.835047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.835021    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6rqw\" (UniqueName: \"kubernetes.io/projected/b07db380-14ed-49cb-b871-ef22cdee2cf8-kube-api-access-m6rqw\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.835047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.835030    4827 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-kubeconfig\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.835047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.835039    4827 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-cacerts\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.835047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:51.835047    4827 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b07db380-14ed-49cb-b871-ef22cdee2cf8-istio-csr-dns-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:46:52.306388 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.306354    4827 generic.go:356] "Generic (PLEG): container finished" podID="b07db380-14ed-49cb-b871-ef22cdee2cf8" containerID="8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b" exitCode=0
Oct 29 22:46:52.306776 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.306420    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"
Oct 29 22:46:52.306776 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.306440    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7" event={"ID":"b07db380-14ed-49cb-b871-ef22cdee2cf8","Type":"ContainerDied","Data":"8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b"}
Oct 29 22:46:52.306776 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.306525    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7" event={"ID":"b07db380-14ed-49cb-b871-ef22cdee2cf8","Type":"ContainerDied","Data":"fb7560696e3707af293971b6550194d2f1c1dd62a5b0d685d9c1944262b41c13"}
Oct 29 22:46:52.306776 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.306551    4827 scope.go:122] "RemoveContainer" containerID="8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b"
Oct 29 22:46:52.315025 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.315007    4827 scope.go:122] "RemoveContainer" containerID="8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b"
Oct 29 22:46:52.315282 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:46:52.315260    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b\": container with ID starting with 8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b not found: ID does not exist" containerID="8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b"
Oct 29 22:46:52.315334 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.315290    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b"} err="failed to get container status \"8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b\": rpc error: code = NotFound desc = could not find container \"8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b\": container with ID starting with 8f912d4108fe87f68e908add48366df6be00b2dfd1220d59e52c30682ae0a19b not found: ID does not exist"
Oct 29 22:46:52.366531 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.366506    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"]
Oct 29 22:46:52.378160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.378138    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lb5s7"]
Oct 29 22:46:52.485041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:52.485001    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07db380-14ed-49cb-b871-ef22cdee2cf8" path="/var/lib/kubelet/pods/b07db380-14ed-49cb-b871-ef22cdee2cf8/volumes"
Oct 29 22:46:55.153135 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.153101    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"]
Oct 29 22:46:55.153517 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.153395    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b07db380-14ed-49cb-b871-ef22cdee2cf8" containerName="discovery"
Oct 29 22:46:55.153517 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.153405    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07db380-14ed-49cb-b871-ef22cdee2cf8" containerName="discovery"
Oct 29 22:46:55.153517 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.153451    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07db380-14ed-49cb-b871-ef22cdee2cf8" containerName="discovery"
Oct 29 22:46:55.157664 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.157644    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.177935 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.177901    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"]
Oct 29 22:46:55.185189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.185169    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-v24ck\""
Oct 29 22:46:55.185314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.185166    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:46:55.185314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.185182    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 29 22:46:55.185461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.185217    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:46:55.259480 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.259450    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e3ce4d59-d3fa-421f-979c-1fcc26069beb-data\") pod \"postgres-7cdbf6dc44-4dzn6\" (UID: \"e3ce4d59-d3fa-421f-979c-1fcc26069beb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.259480 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.259484    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpljn\" (UniqueName: \"kubernetes.io/projected/e3ce4d59-d3fa-421f-979c-1fcc26069beb-kube-api-access-lpljn\") pod \"postgres-7cdbf6dc44-4dzn6\" (UID: \"e3ce4d59-d3fa-421f-979c-1fcc26069beb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.360753 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.360722    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e3ce4d59-d3fa-421f-979c-1fcc26069beb-data\") pod \"postgres-7cdbf6dc44-4dzn6\" (UID: \"e3ce4d59-d3fa-421f-979c-1fcc26069beb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.360753 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.360755    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpljn\" (UniqueName: \"kubernetes.io/projected/e3ce4d59-d3fa-421f-979c-1fcc26069beb-kube-api-access-lpljn\") pod \"postgres-7cdbf6dc44-4dzn6\" (UID: \"e3ce4d59-d3fa-421f-979c-1fcc26069beb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.361112 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.361088    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e3ce4d59-d3fa-421f-979c-1fcc26069beb-data\") pod \"postgres-7cdbf6dc44-4dzn6\" (UID: \"e3ce4d59-d3fa-421f-979c-1fcc26069beb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.383484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.383456    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpljn\" (UniqueName: \"kubernetes.io/projected/e3ce4d59-d3fa-421f-979c-1fcc26069beb-kube-api-access-lpljn\") pod \"postgres-7cdbf6dc44-4dzn6\" (UID: \"e3ce4d59-d3fa-421f-979c-1fcc26069beb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.469761 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.469688    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:46:55.761367 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:55.761291    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"]
Oct 29 22:46:55.765538 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:46:55.765508    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ce4d59_d3fa_421f_979c_1fcc26069beb.slice/crio-10655adbe33e981ce41c399266bb5bce22ed85ea2a43426e013a9f9db6754cac WatchSource:0}: Error finding container 10655adbe33e981ce41c399266bb5bce22ed85ea2a43426e013a9f9db6754cac: Status 404 returned error can't find the container with id 10655adbe33e981ce41c399266bb5bce22ed85ea2a43426e013a9f9db6754cac
Oct 29 22:46:56.326288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:46:56.326250    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6" event={"ID":"e3ce4d59-d3fa-421f-979c-1fcc26069beb","Type":"ContainerStarted","Data":"10655adbe33e981ce41c399266bb5bce22ed85ea2a43426e013a9f9db6754cac"}
Oct 29 22:47:01.345750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:01.345704    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6" event={"ID":"e3ce4d59-d3fa-421f-979c-1fcc26069beb","Type":"ContainerStarted","Data":"b8660104fcafe2a93948da69e0a9c4c7838b4454613149372eea1f33f3626445"}
Oct 29 22:47:01.346116 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:01.345893    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:47:01.385729 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:01.385637    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6" podStartSLOduration=1.019962852 podStartE2EDuration="6.385620135s" totalImagesPullingTime="5.365657283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:01.384690174 +0000 UTC m=+257.007601193" watchObservedRunningTime="2026-10-29 22:47:01.385620135 +0000 UTC m=+257.008531157"
Oct 29 22:47:02.821821 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:47:02.821789    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:47:07.377183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:07.377151    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4dzn6"
Oct 29 22:47:15.203351 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.203311    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b677596ff-6q8cd"]
Oct 29 22:47:15.205629 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.205606    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.219949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.219929    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 29 22:47:15.220065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.219964    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 29 22:47:15.231040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.231020    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9tnk6\""
Oct 29 22:47:15.231240 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.231072    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 29 22:47:15.232142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.232121    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b677596ff-6q8cd"]
Oct 29 22:47:15.316304 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.316271    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-webhook-cert\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.316304 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.316305    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-metrics-certs\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.316526 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.316340    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8fh\" (UniqueName: \"kubernetes.io/projected/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-kube-api-access-sd8fh\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.417060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.417025    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-webhook-cert\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.417249 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.417064    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-metrics-certs\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.417249 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.417099    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8fh\" (UniqueName: \"kubernetes.io/projected/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-kube-api-access-sd8fh\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.419484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.419462    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-metrics-certs\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.419563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.419487    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-webhook-cert\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.440151 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.440123    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8fh\" (UniqueName: \"kubernetes.io/projected/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-kube-api-access-sd8fh\") pod \"maas-controller-5b677596ff-6q8cd\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") " pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.516086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.516008    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:15.741356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.741326    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67b77b449f-756q9"]
Oct 29 22:47:15.744099 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.744079    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.766356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.766299    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b77b449f-756q9"]
Oct 29 22:47:15.820425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.820382    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-metrics-certs\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.820574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.820442    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-webhook-cert\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.820636 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.820566    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdbm\" (UniqueName: \"kubernetes.io/projected/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-kube-api-access-dzdbm\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.821084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.821061    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b677596ff-6q8cd"]
Oct 29 22:47:15.824294 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:47:15.824269    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dcc1d5_51ed_48c3_beed_b5c8a56f1a92.slice/crio-88576d57c9e5a958c07c9e5aa9a166dbb9a4561f406878c3c73910f956912298 WatchSource:0}: Error finding container 88576d57c9e5a958c07c9e5aa9a166dbb9a4561f406878c3c73910f956912298: Status 404 returned error can't find the container with id 88576d57c9e5a958c07c9e5aa9a166dbb9a4561f406878c3c73910f956912298
Oct 29 22:47:15.920987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.920959    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdbm\" (UniqueName: \"kubernetes.io/projected/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-kube-api-access-dzdbm\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.921119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.920996    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-metrics-certs\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.921119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.921020    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-webhook-cert\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.923436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.923414    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-webhook-cert\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.923535 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.923480    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-metrics-certs\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:15.943121 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:15.943100    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdbm\" (UniqueName: \"kubernetes.io/projected/0bcbd4d5-eb73-4d95-8d90-4f91a8379add-kube-api-access-dzdbm\") pod \"maas-controller-67b77b449f-756q9\" (UID: \"0bcbd4d5-eb73-4d95-8d90-4f91a8379add\") " pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:16.055685 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:16.055608    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:16.351562 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:16.351491    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b77b449f-756q9"]
Oct 29 22:47:16.355327 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:47:16.355294    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcbd4d5_eb73_4d95_8d90_4f91a8379add.slice/crio-8268a6fb7a7456b47f8757ce17decd6c1f7eeea0564dbf98a644ae282ca40d04 WatchSource:0}: Error finding container 8268a6fb7a7456b47f8757ce17decd6c1f7eeea0564dbf98a644ae282ca40d04: Status 404 returned error can't find the container with id 8268a6fb7a7456b47f8757ce17decd6c1f7eeea0564dbf98a644ae282ca40d04
Oct 29 22:47:16.398132 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:16.398098    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b77b449f-756q9" event={"ID":"0bcbd4d5-eb73-4d95-8d90-4f91a8379add","Type":"ContainerStarted","Data":"8268a6fb7a7456b47f8757ce17decd6c1f7eeea0564dbf98a644ae282ca40d04"}
Oct 29 22:47:16.399404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:16.399375    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b677596ff-6q8cd" event={"ID":"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92","Type":"ContainerStarted","Data":"88576d57c9e5a958c07c9e5aa9a166dbb9a4561f406878c3c73910f956912298"}
Oct 29 22:47:19.416959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:19.416917    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b77b449f-756q9" event={"ID":"0bcbd4d5-eb73-4d95-8d90-4f91a8379add","Type":"ContainerStarted","Data":"97a0646f6eeefe69ff340905865fe2c35bc7ee483bd9fb6457f3b121c1aa8d52"}
Oct 29 22:47:19.417414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:19.417021    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:19.418370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:19.418348    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b677596ff-6q8cd" event={"ID":"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92","Type":"ContainerStarted","Data":"0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825"}
Oct 29 22:47:19.418529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:19.418506    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:19.464221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:19.464168    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67b77b449f-756q9" podStartSLOduration=1.953234774 podStartE2EDuration="4.464154851s" totalImagesPullingTime="2.510920077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:19.463576903 +0000 UTC m=+275.086487923" watchObservedRunningTime="2026-10-29 22:47:19.464154851 +0000 UTC m=+275.087066188"
Oct 29 22:47:19.505457 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:19.505408    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5b677596ff-6q8cd" podStartSLOduration=1.8049429259999998 podStartE2EDuration="4.505398362s" totalImagesPullingTime="2.700455436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:19.504526149 +0000 UTC m=+275.127437169" watchObservedRunningTime="2026-10-29 22:47:19.505398362 +0000 UTC m=+275.128309382"
Oct 29 22:47:20.584537 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.584507    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"]
Oct 29 22:47:20.588044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.588015    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.605436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.605413    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 29 22:47:20.616424 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.616386    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2p6qk\""
Oct 29 22:47:20.620064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.620041    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"]
Oct 29 22:47:20.641252 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.641229    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 29 22:47:20.641382 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.641264    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 29 22:47:20.766008 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.765979    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e8b81688-d926-4ab2-a36d-5ec456980711-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.766168 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.766024    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8b81688-d926-4ab2-a36d-5ec456980711-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.766168 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.766042    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b81688-d926-4ab2-a36d-5ec456980711-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.766236 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.766158    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwrf\" (UniqueName: \"kubernetes.io/projected/e8b81688-d926-4ab2-a36d-5ec456980711-kube-api-access-6kwrf\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.867001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.866929    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8b81688-d926-4ab2-a36d-5ec456980711-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.867001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.866961    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b81688-d926-4ab2-a36d-5ec456980711-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.867001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.866985    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwrf\" (UniqueName: \"kubernetes.io/projected/e8b81688-d926-4ab2-a36d-5ec456980711-kube-api-access-6kwrf\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.867260 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.867040    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e8b81688-d926-4ab2-a36d-5ec456980711-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.867830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.867805    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b81688-d926-4ab2-a36d-5ec456980711-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.867951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.867935    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e8b81688-d926-4ab2-a36d-5ec456980711-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.869245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.869223    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8b81688-d926-4ab2-a36d-5ec456980711-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.889565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.889534    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwrf\" (UniqueName: \"kubernetes.io/projected/e8b81688-d926-4ab2-a36d-5ec456980711-kube-api-access-6kwrf\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rkptj\" (UID: \"e8b81688-d926-4ab2-a36d-5ec456980711\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:20.900414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:20.900392    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:21.194795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:21.194751    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"]
Oct 29 22:47:21.198085 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:47:21.198058    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b81688_d926_4ab2_a36d_5ec456980711.slice/crio-ec58ded2fea6ad21f9293ee672157b3de60adb822294fcb61be61a88b009d701 WatchSource:0}: Error finding container ec58ded2fea6ad21f9293ee672157b3de60adb822294fcb61be61a88b009d701: Status 404 returned error can't find the container with id ec58ded2fea6ad21f9293ee672157b3de60adb822294fcb61be61a88b009d701
Oct 29 22:47:21.427821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:21.427782    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj" event={"ID":"e8b81688-d926-4ab2-a36d-5ec456980711","Type":"ContainerStarted","Data":"ec58ded2fea6ad21f9293ee672157b3de60adb822294fcb61be61a88b009d701"}
Oct 29 22:47:30.424448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.424415    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:30.425008 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.424989    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67b77b449f-756q9"
Oct 29 22:47:30.544459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.544084    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b677596ff-6q8cd"]
Oct 29 22:47:30.544459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.544278    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5b677596ff-6q8cd" podUID="c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" containerName="manager" containerID="cri-o://0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825" gracePeriod=30
Oct 29 22:47:30.783630 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.783602    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:30.850198 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.850166    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-kube-api-access-sd8fh\" (UniqueName: \"kubernetes.io/projected/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-kube-api-access-sd8fh\") pod \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") "
Oct 29 22:47:30.850331 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.850212    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-metrics-certs\") pod \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") "
Oct 29 22:47:30.852572 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.852542    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-kube-api-access-sd8fh" pod "c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" (UID: "c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92"). InnerVolumeSpecName "kube-api-access-sd8fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:30.852572 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.852565    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-metrics-certs" pod "c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" (UID: "c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:47:30.951447 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.951423    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-webhook-cert\") pod \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\" (UID: \"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92\") "
Oct 29 22:47:30.951569 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.951557    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8fh\" (UniqueName: \"kubernetes.io/projected/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-kube-api-access-sd8fh\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:47:30.951610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.951572    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:47:30.953118 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:30.953093    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-webhook-cert" pod "c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" (UID: "c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:47:31.052065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.051990    4827 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92-webhook-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:47:31.475394 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.475353    4827 generic.go:356] "Generic (PLEG): container finished" podID="c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" containerID="0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825" exitCode=0
Oct 29 22:47:31.476036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.475403    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b677596ff-6q8cd"
Oct 29 22:47:31.476036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.475443    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b677596ff-6q8cd" event={"ID":"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92","Type":"ContainerDied","Data":"0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825"}
Oct 29 22:47:31.476036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.475500    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b677596ff-6q8cd" event={"ID":"c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92","Type":"ContainerDied","Data":"88576d57c9e5a958c07c9e5aa9a166dbb9a4561f406878c3c73910f956912298"}
Oct 29 22:47:31.476036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.475529    4827 scope.go:122] "RemoveContainer" containerID="0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825"
Oct 29 22:47:31.477021 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.476995    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj" event={"ID":"e8b81688-d926-4ab2-a36d-5ec456980711","Type":"ContainerStarted","Data":"55eecda2aee053af2df009cb86678974a0f7c55cb5685617225ed7e9894aea70"}
Oct 29 22:47:31.477126 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.477051    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:31.484104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.484089    4827 scope.go:122] "RemoveContainer" containerID="0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825"
Oct 29 22:47:31.484351 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:47:31.484333    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825\": container with ID starting with 0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825 not found: ID does not exist" containerID="0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825"
Oct 29 22:47:31.484409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.484359    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825"} err="failed to get container status \"0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825\": rpc error: code = NotFound desc = could not find container \"0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825\": container with ID starting with 0019510193dc9b46293e14dfbe78c5d63bab26d7a120f5ad10825a2fea25d825 not found: ID does not exist"
Oct 29 22:47:31.533822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.533745    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj" podStartSLOduration=1.840629098 podStartE2EDuration="11.533729757s" totalImagesPullingTime="9.693100659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:31.532598397 +0000 UTC m=+287.155509419" watchObservedRunningTime="2026-10-29 22:47:31.533729757 +0000 UTC m=+287.156640777"
Oct 29 22:47:31.567626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.567590    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b677596ff-6q8cd"]
Oct 29 22:47:31.573794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:31.573402    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b677596ff-6q8cd"]
Oct 29 22:47:32.485159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:32.485131    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" path="/var/lib/kubelet/pods/c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92/volumes"
Oct 29 22:47:32.821278 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:47:32.821204    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:47:42.488051 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:42.488016    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rkptj"
Oct 29 22:47:44.504664 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:44.504643    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:47:44.505337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:47:44.505316    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:48:01.918239 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:01.918165    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2slkw"]
Oct 29 22:48:01.918840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:01.918532    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" containerName="manager"
Oct 29 22:48:01.918840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:01.918550    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" containerName="manager"
Oct 29 22:48:01.918840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:01.918624    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2dcc1d5-51ed-48c3-beed-b5c8a56f1a92" containerName="manager"
Oct 29 22:48:02.007757 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.007721    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2slkw"]
Oct 29 22:48:02.007919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.007833    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.034562 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.034531    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ppj7r\""
Oct 29 22:48:02.034562 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.034531    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 29 22:48:02.035598 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.035574    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2slkw"]
Oct 29 22:48:02.062820 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:48:02.062786    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2gtn4 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-2slkw" podUID="1ba435aa-8834-4b2d-bb5c-34b36363134b"
Oct 29 22:48:02.085093 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.085064    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba435aa-8834-4b2d-bb5c-34b36363134b-tls-cert\") pod \"authorino-6889cbd8c4-2slkw\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") " pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.085203 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.085101    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtn4\" (UniqueName: \"kubernetes.io/projected/1ba435aa-8834-4b2d-bb5c-34b36363134b-kube-api-access-2gtn4\") pod \"authorino-6889cbd8c4-2slkw\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") " pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.186297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.186216    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba435aa-8834-4b2d-bb5c-34b36363134b-tls-cert\") pod \"authorino-6889cbd8c4-2slkw\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") " pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.186297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.186262    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtn4\" (UniqueName: \"kubernetes.io/projected/1ba435aa-8834-4b2d-bb5c-34b36363134b-kube-api-access-2gtn4\") pod \"authorino-6889cbd8c4-2slkw\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") " pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.188717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.188693    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba435aa-8834-4b2d-bb5c-34b36363134b-tls-cert\") pod \"authorino-6889cbd8c4-2slkw\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") " pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.210288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.210262    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtn4\" (UniqueName: \"kubernetes.io/projected/1ba435aa-8834-4b2d-bb5c-34b36363134b-kube-api-access-2gtn4\") pod \"authorino-6889cbd8c4-2slkw\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") " pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.582580 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.582501    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.633425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.633386    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:02.690536 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.690496    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba435aa-8834-4b2d-bb5c-34b36363134b-kube-api-access-2gtn4\" (UniqueName: \"kubernetes.io/projected/1ba435aa-8834-4b2d-bb5c-34b36363134b-kube-api-access-2gtn4\") pod \"1ba435aa-8834-4b2d-bb5c-34b36363134b\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") "
Oct 29 22:48:02.690726 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.690553    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ba435aa-8834-4b2d-bb5c-34b36363134b-tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba435aa-8834-4b2d-bb5c-34b36363134b-tls-cert\") pod \"1ba435aa-8834-4b2d-bb5c-34b36363134b\" (UID: \"1ba435aa-8834-4b2d-bb5c-34b36363134b\") "
Oct 29 22:48:02.692635 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.692609    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba435aa-8834-4b2d-bb5c-34b36363134b-tls-cert" pod "1ba435aa-8834-4b2d-bb5c-34b36363134b" (UID: "1ba435aa-8834-4b2d-bb5c-34b36363134b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:02.692752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.692639    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba435aa-8834-4b2d-bb5c-34b36363134b-kube-api-access-2gtn4" pod "1ba435aa-8834-4b2d-bb5c-34b36363134b" (UID: "1ba435aa-8834-4b2d-bb5c-34b36363134b"). InnerVolumeSpecName "kube-api-access-2gtn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:02.791315 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.791277    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gtn4\" (UniqueName: \"kubernetes.io/projected/1ba435aa-8834-4b2d-bb5c-34b36363134b-kube-api-access-2gtn4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:48:02.791315 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:02.791308    4827 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba435aa-8834-4b2d-bb5c-34b36363134b-tls-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:48:02.821376 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:48:02.821341    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:48:03.585376 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:03.585342    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2slkw"
Oct 29 22:48:03.661183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:03.661152    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2slkw"]
Oct 29 22:48:03.663209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:03.663187    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2slkw"]
Oct 29 22:48:04.485036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:04.485001    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba435aa-8834-4b2d-bb5c-34b36363134b" path="/var/lib/kubelet/pods/1ba435aa-8834-4b2d-bb5c-34b36363134b/volumes"
Oct 29 22:48:32.821524 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:48:32.821482    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:48:39.270060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.270027    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-28zgn"]
Oct 29 22:48:39.273816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.273794    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.301578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.301552    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mg94d\""
Oct 29 22:48:39.301578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.301552    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 29 22:48:39.301824 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.301809    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 29 22:48:39.303172 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.303147    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-28zgn"]
Oct 29 22:48:39.380456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.380426    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59sn\" (UniqueName: \"kubernetes.io/projected/ef9c885c-a06f-40a9-a544-eae42fa50b0d-kube-api-access-q59sn\") pod \"ai-gateway-controller-7d85675f98-28zgn\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.380456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.380458    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9c885c-a06f-40a9-a544-eae42fa50b0d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-28zgn\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.481170 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.481138    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59sn\" (UniqueName: \"kubernetes.io/projected/ef9c885c-a06f-40a9-a544-eae42fa50b0d-kube-api-access-q59sn\") pod \"ai-gateway-controller-7d85675f98-28zgn\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.481170 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.481171    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9c885c-a06f-40a9-a544-eae42fa50b0d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-28zgn\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.483590 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.483565    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9c885c-a06f-40a9-a544-eae42fa50b0d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-28zgn\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.504689 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.504657    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59sn\" (UniqueName: \"kubernetes.io/projected/ef9c885c-a06f-40a9-a544-eae42fa50b0d-kube-api-access-q59sn\") pod \"ai-gateway-controller-7d85675f98-28zgn\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.585740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.585657    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:39.875549 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:39.875380    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-28zgn"]
Oct 29 22:48:39.878495 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:48:39.878466    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9c885c_a06f_40a9_a544_eae42fa50b0d.slice/crio-2d9c3c9bc8dc1997ab875fe46b89052aaef6cf6279ee01eaa1d4c06014f56236 WatchSource:0}: Error finding container 2d9c3c9bc8dc1997ab875fe46b89052aaef6cf6279ee01eaa1d4c06014f56236: Status 404 returned error can't find the container with id 2d9c3c9bc8dc1997ab875fe46b89052aaef6cf6279ee01eaa1d4c06014f56236
Oct 29 22:48:40.713138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:40.713102    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn" event={"ID":"ef9c885c-a06f-40a9-a544-eae42fa50b0d","Type":"ContainerStarted","Data":"2d9c3c9bc8dc1997ab875fe46b89052aaef6cf6279ee01eaa1d4c06014f56236"}
Oct 29 22:48:41.718138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:41.718099    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn" event={"ID":"ef9c885c-a06f-40a9-a544-eae42fa50b0d","Type":"ContainerStarted","Data":"64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e"}
Oct 29 22:48:41.718591 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:41.718196    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:41.760103 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:41.760057    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn" podStartSLOduration=1.280490814 podStartE2EDuration="2.760045317s" totalImagesPullingTime="1.479554503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:41.759575384 +0000 UTC m=+357.382486403" watchObservedRunningTime="2026-10-29 22:48:41.760045317 +0000 UTC m=+357.382956339"
Oct 29 22:48:43.555263 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.555229    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-tjghh"]
Oct 29 22:48:43.558622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.558598    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:43.578954 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.578933    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 29 22:48:43.591976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.591951    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-tjghh"]
Oct 29 22:48:43.618875 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.618845    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrk24\" (UniqueName: \"kubernetes.io/projected/8160871b-24d7-4b6b-bbd5-64ed33ae6382-kube-api-access-jrk24\") pod \"payload-processing-5f78687d5b-tjghh\" (UID: \"8160871b-24d7-4b6b-bbd5-64ed33ae6382\") " pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:43.619010 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.618890    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8160871b-24d7-4b6b-bbd5-64ed33ae6382-plugins-config-volume\") pod \"payload-processing-5f78687d5b-tjghh\" (UID: \"8160871b-24d7-4b6b-bbd5-64ed33ae6382\") " pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:43.719682 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.719650    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8160871b-24d7-4b6b-bbd5-64ed33ae6382-plugins-config-volume\") pod \"payload-processing-5f78687d5b-tjghh\" (UID: \"8160871b-24d7-4b6b-bbd5-64ed33ae6382\") " pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:43.719875 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.719725    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrk24\" (UniqueName: \"kubernetes.io/projected/8160871b-24d7-4b6b-bbd5-64ed33ae6382-kube-api-access-jrk24\") pod \"payload-processing-5f78687d5b-tjghh\" (UID: \"8160871b-24d7-4b6b-bbd5-64ed33ae6382\") " pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:43.720441 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.720416    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8160871b-24d7-4b6b-bbd5-64ed33ae6382-plugins-config-volume\") pod \"payload-processing-5f78687d5b-tjghh\" (UID: \"8160871b-24d7-4b6b-bbd5-64ed33ae6382\") " pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:43.743047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.743019    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrk24\" (UniqueName: \"kubernetes.io/projected/8160871b-24d7-4b6b-bbd5-64ed33ae6382-kube-api-access-jrk24\") pod \"payload-processing-5f78687d5b-tjghh\" (UID: \"8160871b-24d7-4b6b-bbd5-64ed33ae6382\") " pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:43.871283 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:43.871199    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:44.164246 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:44.164216    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-tjghh"]
Oct 29 22:48:44.167363 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:48:44.167341    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8160871b_24d7_4b6b_bbd5_64ed33ae6382.slice/crio-a7905fc2cb6325ebb8d0a69929fdf9c7d57eed501f80ec3b3ca1347e58fb71a6 WatchSource:0}: Error finding container a7905fc2cb6325ebb8d0a69929fdf9c7d57eed501f80ec3b3ca1347e58fb71a6: Status 404 returned error can't find the container with id a7905fc2cb6325ebb8d0a69929fdf9c7d57eed501f80ec3b3ca1347e58fb71a6
Oct 29 22:48:44.734861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:44.734817    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-tjghh" event={"ID":"8160871b-24d7-4b6b-bbd5-64ed33ae6382","Type":"ContainerStarted","Data":"a7905fc2cb6325ebb8d0a69929fdf9c7d57eed501f80ec3b3ca1347e58fb71a6"}
Oct 29 22:48:46.748696 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:46.744690    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-tjghh" event={"ID":"8160871b-24d7-4b6b-bbd5-64ed33ae6382","Type":"ContainerStarted","Data":"1cbbc3357ec3f04c7654a3a702be430effd9f75bd779160efc3cbce40d4c700f"}
Oct 29 22:48:46.748696 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:46.745406    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:48:46.785601 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:46.785553    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-tjghh" podStartSLOduration=1.76578605 podStartE2EDuration="3.785539632s" totalImagesPullingTime="2.019753582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:46.784415055 +0000 UTC m=+362.407326074" watchObservedRunningTime="2026-10-29 22:48:46.785539632 +0000 UTC m=+362.408450649"
Oct 29 22:48:52.724910 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:52.724874    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:48:57.751459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:48:57.751427    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-tjghh"
Oct 29 22:49:02.822038 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:49:02.821999    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:49:06.233963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.233926    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-28zgn"]
Oct 29 22:49:06.234436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.234162    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn" podUID="ef9c885c-a06f-40a9-a544-eae42fa50b0d" containerName="manager" containerID="cri-o://64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e" gracePeriod=30
Oct 29 22:49:06.467514 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.467490    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:49:06.481717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.481694    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9c885c-a06f-40a9-a544-eae42fa50b0d-kube-api-access-q59sn\" (UniqueName: \"kubernetes.io/projected/ef9c885c-a06f-40a9-a544-eae42fa50b0d-kube-api-access-q59sn\") pod \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") "
Oct 29 22:49:06.481822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.481761    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9c885c-a06f-40a9-a544-eae42fa50b0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9c885c-a06f-40a9-a544-eae42fa50b0d-metrics-certs\") pod \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\" (UID: \"ef9c885c-a06f-40a9-a544-eae42fa50b0d\") "
Oct 29 22:49:06.483758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.483732    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9c885c-a06f-40a9-a544-eae42fa50b0d-metrics-certs" pod "ef9c885c-a06f-40a9-a544-eae42fa50b0d" (UID: "ef9c885c-a06f-40a9-a544-eae42fa50b0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:06.483870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.483814    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9c885c-a06f-40a9-a544-eae42fa50b0d-kube-api-access-q59sn" pod "ef9c885c-a06f-40a9-a544-eae42fa50b0d" (UID: "ef9c885c-a06f-40a9-a544-eae42fa50b0d"). InnerVolumeSpecName "kube-api-access-q59sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:06.582847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.582814    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59sn\" (UniqueName: \"kubernetes.io/projected/ef9c885c-a06f-40a9-a544-eae42fa50b0d-kube-api-access-q59sn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:49:06.582847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.582842    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9c885c-a06f-40a9-a544-eae42fa50b0d-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:49:06.819604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.819515    4827 generic.go:356] "Generic (PLEG): container finished" podID="ef9c885c-a06f-40a9-a544-eae42fa50b0d" containerID="64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e" exitCode=0
Oct 29 22:49:06.819604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.819563    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn"
Oct 29 22:49:06.819803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.819565    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn" event={"ID":"ef9c885c-a06f-40a9-a544-eae42fa50b0d","Type":"ContainerDied","Data":"64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e"}
Oct 29 22:49:06.819803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.819691    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-28zgn" event={"ID":"ef9c885c-a06f-40a9-a544-eae42fa50b0d","Type":"ContainerDied","Data":"2d9c3c9bc8dc1997ab875fe46b89052aaef6cf6279ee01eaa1d4c06014f56236"}
Oct 29 22:49:06.819803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.819719    4827 scope.go:122] "RemoveContainer" containerID="64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e"
Oct 29 22:49:06.827318 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.827293    4827 scope.go:122] "RemoveContainer" containerID="64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e"
Oct 29 22:49:06.827584 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:49:06.827564    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e\": container with ID starting with 64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e not found: ID does not exist" containerID="64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e"
Oct 29 22:49:06.827644 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.827593    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e"} err="failed to get container status \"64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e\": rpc error: code = NotFound desc = could not find container \"64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e\": container with ID starting with 64fab4bce60b4ca2bb71e36479a71a4ae9088e9adb820f201d296602879c381e not found: ID does not exist"
Oct 29 22:49:06.869593 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.869564    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-28zgn"]
Oct 29 22:49:06.881494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:06.881465    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-28zgn"]
Oct 29 22:49:08.485084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:08.485054    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9c885c-a06f-40a9-a544-eae42fa50b0d" path="/var/lib/kubelet/pods/ef9c885c-a06f-40a9-a544-eae42fa50b0d/volumes"
Oct 29 22:49:17.606553 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.606518    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"]
Oct 29 22:49:17.607117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.607014    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ef9c885c-a06f-40a9-a544-eae42fa50b0d" containerName="manager"
Oct 29 22:49:17.607117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.607033    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9c885c-a06f-40a9-a544-eae42fa50b0d" containerName="manager"
Oct 29 22:49:17.607226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.607117    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9c885c-a06f-40a9-a544-eae42fa50b0d" containerName="manager"
Oct 29 22:49:17.615835 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.615805    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.632659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.632633    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"]
Oct 29 22:49:17.643843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.643817    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8dzpq\""
Oct 29 22:49:17.643843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.643830    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 22:49:17.643993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.643817    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 22:49:17.643993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.643839    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 29 22:49:17.670982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.670956    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8n7\" (UniqueName: \"kubernetes.io/projected/89beb379-c7e3-441a-90cb-5572b7d934da-kube-api-access-sk8n7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.671111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.670992    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.671111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.671011    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.671111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.671032    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.671266 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.671144    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89beb379-c7e3-441a-90cb-5572b7d934da-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.671266 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.671237    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.771982    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772172 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772033    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8n7\" (UniqueName: \"kubernetes.io/projected/89beb379-c7e3-441a-90cb-5572b7d934da-kube-api-access-sk8n7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772172 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772055    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772172 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772157    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772341 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772210    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772341 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772238    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89beb379-c7e3-441a-90cb-5572b7d934da-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772476 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772458    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772540 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772523    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.772642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.772620    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.774414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.774391    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89beb379-c7e3-441a-90cb-5572b7d934da-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.774609 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.774592    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89beb379-c7e3-441a-90cb-5572b7d934da-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.795256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.795230    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8n7\" (UniqueName: \"kubernetes.io/projected/89beb379-c7e3-441a-90cb-5572b7d934da-kube-api-access-sk8n7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c874f\" (UID: \"89beb379-c7e3-441a-90cb-5572b7d934da\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:17.933297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:17.933262    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:18.226688 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:18.226658    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"]
Oct 29 22:49:18.229341 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:49:18.229308    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89beb379_c7e3_441a_90cb_5572b7d934da.slice/crio-399f210766007fad995ce1c2bc46977a17cc307cff303ffb01b89d6bb4069030 WatchSource:0}: Error finding container 399f210766007fad995ce1c2bc46977a17cc307cff303ffb01b89d6bb4069030: Status 404 returned error can't find the container with id 399f210766007fad995ce1c2bc46977a17cc307cff303ffb01b89d6bb4069030
Oct 29 22:49:18.865439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:18.865400    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f" event={"ID":"89beb379-c7e3-441a-90cb-5572b7d934da","Type":"ContainerStarted","Data":"399f210766007fad995ce1c2bc46977a17cc307cff303ffb01b89d6bb4069030"}
Oct 29 22:49:19.870487 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:19.870450    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f" event={"ID":"89beb379-c7e3-441a-90cb-5572b7d934da","Type":"ContainerStarted","Data":"2af379179459a6e4e8b62a607e7bf20234079df94cf9449340357e77e7b08930"}
Oct 29 22:49:19.911984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:19.911931    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f" podStartSLOduration=1.907766397 podStartE2EDuration="2.911917044s" totalImagesPullingTime="1.004150647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:19.911728305 +0000 UTC m=+395.534639327" watchObservedRunningTime="2026-10-29 22:49:19.911917044 +0000 UTC m=+395.534828057"
Oct 29 22:49:19.933804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:19.933749    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:21.412957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.412924    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"]
Oct 29 22:49:21.416614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.416591    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.443293 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.443270    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-external-token-limit-simulated-kserve-self-signed-certs\""
Oct 29 22:49:21.444675 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.444653    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"]
Oct 29 22:49:21.504823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.504786    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.504976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.504873    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.504976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.504922    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.505052 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.504970    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nggh\" (UniqueName: \"kubernetes.io/projected/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-kube-api-access-6nggh\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.505052 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.505000    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.505052 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.505040    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606351    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606637 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606422    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606855    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606858    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606886    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nggh\" (UniqueName: \"kubernetes.io/projected/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-kube-api-access-6nggh\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606902    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606927    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.606976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.606967    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.610280 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.610251    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.610492 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.610469    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.629894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.629861    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nggh\" (UniqueName: \"kubernetes.io/projected/f90fd47c-4d11-4faa-8eb9-d394960cc2cf-kube-api-access-6nggh\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x\" (UID: \"f90fd47c-4d11-4faa-8eb9-d394960cc2cf\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:21.730216 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:21.730135    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:22.027296 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:22.027066    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"]
Oct 29 22:49:22.029805 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:49:22.029775    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90fd47c_4d11_4faa_8eb9_d394960cc2cf.slice/crio-787e6cf176c4a5bd6848fb2f073d192c61687ac0366fd3b54f508e8610273392 WatchSource:0}: Error finding container 787e6cf176c4a5bd6848fb2f073d192c61687ac0366fd3b54f508e8610273392: Status 404 returned error can't find the container with id 787e6cf176c4a5bd6848fb2f073d192c61687ac0366fd3b54f508e8610273392
Oct 29 22:49:22.883184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:22.883149    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x" event={"ID":"f90fd47c-4d11-4faa-8eb9-d394960cc2cf","Type":"ContainerStarted","Data":"7895c37c3653080f23efb4b3849a5a4486c5f43d726fc6a6b65bfb0c78a1c456"}
Oct 29 22:49:22.883564 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:22.883228    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x" event={"ID":"f90fd47c-4d11-4faa-8eb9-d394960cc2cf","Type":"ContainerStarted","Data":"787e6cf176c4a5bd6848fb2f073d192c61687ac0366fd3b54f508e8610273392"}
Oct 29 22:49:22.924573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:22.924525    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x" podStartSLOduration=1.726811076 podStartE2EDuration="1.924513558s" totalImagesPullingTime="197.702482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:22.924040309 +0000 UTC m=+398.546951329" watchObservedRunningTime="2026-10-29 22:49:22.924513558 +0000 UTC m=+398.547424577"
Oct 29 22:49:23.730404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:23.730372    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:25.741858 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.741822    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"]
Oct 29 22:49:25.745318 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.745297    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.760325 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.760304    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 29 22:49:25.773575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.773550    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"]
Oct 29 22:49:25.844405 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.844371    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.844405 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.844416    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkms\" (UniqueName: \"kubernetes.io/projected/d8d66d1c-4fef-4374-8b11-cf82ad34a314-kube-api-access-gzkms\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.844621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.844464    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d66d1c-4fef-4374-8b11-cf82ad34a314-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.844621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.844546    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.844621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.844567    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.844621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.844592    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.945737 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.945709    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.945927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.945751    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.945927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.945814    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.945927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.945887    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.945927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.945910    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkms\" (UniqueName: \"kubernetes.io/projected/d8d66d1c-4fef-4374-8b11-cf82ad34a314-kube-api-access-gzkms\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.946151 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.945950    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d66d1c-4fef-4374-8b11-cf82ad34a314-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.946241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.946217    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.946241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.946233    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.946642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.946614    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.948302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.948271    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d66d1c-4fef-4374-8b11-cf82ad34a314-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.948984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.948960    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d66d1c-4fef-4374-8b11-cf82ad34a314-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:25.969278 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:25.969255    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkms\" (UniqueName: \"kubernetes.io/projected/d8d66d1c-4fef-4374-8b11-cf82ad34a314-kube-api-access-gzkms\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8\" (UID: \"d8d66d1c-4fef-4374-8b11-cf82ad34a314\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:26.060467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:26.060394    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:26.352132 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:26.352055    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"]
Oct 29 22:49:26.355219 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:49:26.355187    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d66d1c_4fef_4374_8b11_cf82ad34a314.slice/crio-8ae3a8bdb795bf7c21e9479fb55a72d6b3004c280d57393e10e41fc7ec463b08 WatchSource:0}: Error finding container 8ae3a8bdb795bf7c21e9479fb55a72d6b3004c280d57393e10e41fc7ec463b08: Status 404 returned error can't find the container with id 8ae3a8bdb795bf7c21e9479fb55a72d6b3004c280d57393e10e41fc7ec463b08
Oct 29 22:49:26.900755 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:26.900718    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8" event={"ID":"d8d66d1c-4fef-4374-8b11-cf82ad34a314","Type":"ContainerStarted","Data":"1b78851a7965871b4b65c0c526c2757b0165d45ae28f257ba4b2e8661fb35aeb"}
Oct 29 22:49:26.901153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:26.900799    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8" event={"ID":"d8d66d1c-4fef-4374-8b11-cf82ad34a314","Type":"ContainerStarted","Data":"8ae3a8bdb795bf7c21e9479fb55a72d6b3004c280d57393e10e41fc7ec463b08"}
Oct 29 22:49:26.943052 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:26.942976    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8" podStartSLOduration=1.7451593079999999 podStartE2EDuration="1.942963198s" totalImagesPullingTime="197.80389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:26.942082743 +0000 UTC m=+402.564993763" watchObservedRunningTime="2026-10-29 22:49:26.942963198 +0000 UTC m=+402.565874235"
Oct 29 22:49:27.061370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:27.061344    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:27.933957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:27.933920    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:27.946101 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:27.946076    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:28.929443 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:28.929418    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f"
Oct 29 22:49:31.160955 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.160923    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"]
Oct 29 22:49:31.165054 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.165031    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.196208 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.196176    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 29 22:49:31.197308 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.197283    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"]
Oct 29 22:49:31.290199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.290167    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.290369 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.290226    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbc053-4c55-47e4-8345-b2e049215d5d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.290369 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.290322    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.290462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.290388    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qf7\" (UniqueName: \"kubernetes.io/projected/fabbc053-4c55-47e4-8345-b2e049215d5d-kube-api-access-s4qf7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.290496 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.290453    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.290496 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.290480    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391222 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391185    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbc053-4c55-47e4-8345-b2e049215d5d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391222 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391229    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391267    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qf7\" (UniqueName: \"kubernetes.io/projected/fabbc053-4c55-47e4-8345-b2e049215d5d-kube-api-access-s4qf7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391286    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391302    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391323    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391775 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391740    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391749    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.391900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.391804    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.393523 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.393501    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbc053-4c55-47e4-8345-b2e049215d5d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.393822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.393805    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbc053-4c55-47e4-8345-b2e049215d5d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.414196 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.414139    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qf7\" (UniqueName: \"kubernetes.io/projected/fabbc053-4c55-47e4-8345-b2e049215d5d-kube-api-access-s4qf7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb\" (UID: \"fabbc053-4c55-47e4-8345-b2e049215d5d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.492421 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.492388    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:31.731273 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.731195    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:31.743355 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.743331    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:31.797857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.797831    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"]
Oct 29 22:49:31.801244 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:49:31.801220    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfabbc053_4c55_47e4_8345_b2e049215d5d.slice/crio-31bed9c98daac82b4019b043632b89a888aeb3b8943d88b6bf428872e040f10c WatchSource:0}: Error finding container 31bed9c98daac82b4019b043632b89a888aeb3b8943d88b6bf428872e040f10c: Status 404 returned error can't find the container with id 31bed9c98daac82b4019b043632b89a888aeb3b8943d88b6bf428872e040f10c
Oct 29 22:49:31.928404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.928369    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" event={"ID":"fabbc053-4c55-47e4-8345-b2e049215d5d","Type":"ContainerStarted","Data":"31bed9c98daac82b4019b043632b89a888aeb3b8943d88b6bf428872e040f10c"}
Oct 29 22:49:31.939453 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:31.939429    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x"
Oct 29 22:49:32.821377 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:49:32.821339    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:49:32.934826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:32.934783    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" event={"ID":"fabbc053-4c55-47e4-8345-b2e049215d5d","Type":"ContainerStarted","Data":"f4eacb2606a9003802f5d0bf301fbac34122ab679bb6b732e7a3ec58493b218e"}
Oct 29 22:49:32.977351 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:32.977295    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podStartSLOduration=1.802959949 podStartE2EDuration="1.977270402s" totalImagesPullingTime="174.310453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:32.975924424 +0000 UTC m=+408.598835445" watchObservedRunningTime="2026-10-29 22:49:32.977270402 +0000 UTC m=+408.600181422"
Oct 29 22:49:33.493302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:33.493271    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:36.060759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:36.060726    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:36.073115 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:36.073095    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:36.961733 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:36.961708    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8"
Oct 29 22:49:40.908569 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:40.908531    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"]
Oct 29 22:49:40.912159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:40.912135    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:40.924789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:40.924749    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"]
Oct 29 22:49:40.928408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:40.928391    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:40.935078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:40.935050    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2tp5f\""
Oct 29 22:49:40.937031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:40.936970    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"]
Oct 29 22:49:40.965847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:40.965464    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"]
Oct 29 22:49:41.075877 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.075848    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.075887    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.075909    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.075950    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.075967    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlrt\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-kube-api-access-hnlrt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076039    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56dd7a0-03e2-4206-a490-555d93824666-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076122    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076147    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076211    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076241    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076259    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56dd7a0-03e2-4206-a490-555d93824666-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076302    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076343    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076432 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076413    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076432    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56dd7a0-03e2-4206-a490-555d93824666-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076460    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgj4\" (UniqueName: \"kubernetes.io/projected/d56dd7a0-03e2-4206-a490-555d93824666-kube-api-access-pdgj4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.076526 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076482    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.076558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.076533    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.177659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177583    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.177659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177615    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.177659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177643    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.177972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177665    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.177972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177681    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56dd7a0-03e2-4206-a490-555d93824666-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.177972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177835    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.177972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177898    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.177972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177966    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.177993    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56dd7a0-03e2-4206-a490-555d93824666-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.178243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178056    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgj4\" (UniqueName: \"kubernetes.io/projected/d56dd7a0-03e2-4206-a490-555d93824666-kube-api-access-pdgj4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.178243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178090    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178132    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178142    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.178243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178217    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178258    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178286    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178304    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178310    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178367    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178372    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlrt\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-kube-api-access-hnlrt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178423    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56dd7a0-03e2-4206-a490-555d93824666-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.178533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178487    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56dd7a0-03e2-4206-a490-555d93824666-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.179001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178552    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.179001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178333    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.179001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178694    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.179001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178744    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.179001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.178991    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.179321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.179300    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.180160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.180132    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.180967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.180943    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.180967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.180963    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56dd7a0-03e2-4206-a490-555d93824666-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.181121 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.181088    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56dd7a0-03e2-4206-a490-555d93824666-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.201250 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.201230    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56dd7a0-03e2-4206-a490-555d93824666-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.201406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.201385    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.201620 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.201602    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgj4\" (UniqueName: \"kubernetes.io/projected/d56dd7a0-03e2-4206-a490-555d93824666-kube-api-access-pdgj4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x7gtt\" (UID: \"d56dd7a0-03e2-4206-a490-555d93824666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.201679 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.201661    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlrt\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-kube-api-access-hnlrt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-4rrmt\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.225352 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.225332    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:41.241581 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.241561    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:41.492991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.492924    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:41.505451 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.505423    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:41.534576 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.534505    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"]
Oct 29 22:49:41.550296 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:49:41.550274    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3419fb5_2606_4c5e_89fd_7efa2ab90437.slice/crio-36c900f656765268fd514fd3f5bda444b858102fb81691064d48d2627ab3addf WatchSource:0}: Error finding container 36c900f656765268fd514fd3f5bda444b858102fb81691064d48d2627ab3addf: Status 404 returned error can't find the container with id 36c900f656765268fd514fd3f5bda444b858102fb81691064d48d2627ab3addf
Oct 29 22:49:41.552390 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.552355    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:41.552493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.552430    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:41.552552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.552500    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:41.575470 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.574573    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"]
Oct 29 22:49:41.580964 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:49:41.580940    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56dd7a0_03e2_4206_a490_555d93824666.slice/crio-56c03fd75a19b4753ec861aa26cf2a601c9d09d9e57e9dafc5690ea140fc9da1 WatchSource:0}: Error finding container 56c03fd75a19b4753ec861aa26cf2a601c9d09d9e57e9dafc5690ea140fc9da1: Status 404 returned error can't find the container with id 56c03fd75a19b4753ec861aa26cf2a601c9d09d9e57e9dafc5690ea140fc9da1
Oct 29 22:49:41.582873 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.582845    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:41.582938 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.582911    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:41.582977 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.582940    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:41.973441 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.973403    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" event={"ID":"f3419fb5-2606-4c5e-89fd-7efa2ab90437","Type":"ContainerStarted","Data":"7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284"}
Oct 29 22:49:41.973963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.973459    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" event={"ID":"f3419fb5-2606-4c5e-89fd-7efa2ab90437","Type":"ContainerStarted","Data":"36c900f656765268fd514fd3f5bda444b858102fb81691064d48d2627ab3addf"}
Oct 29 22:49:41.975134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.975105    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" event={"ID":"d56dd7a0-03e2-4206-a490-555d93824666","Type":"ContainerStarted","Data":"5f0a1f2757777f1124c86bebb72550a99b652ed35a045764406f116868ca5b7a"}
Oct 29 22:49:41.975257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.975162    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" event={"ID":"d56dd7a0-03e2-4206-a490-555d93824666","Type":"ContainerStarted","Data":"56c03fd75a19b4753ec861aa26cf2a601c9d09d9e57e9dafc5690ea140fc9da1"}
Oct 29 22:49:41.986895 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:41.986872    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:49:42.075790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.069221    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podStartSLOduration=2.06920059 podStartE2EDuration="2.06920059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:42.022388759 +0000 UTC m=+417.645299783" watchObservedRunningTime="2026-10-29 22:49:42.06920059 +0000 UTC m=+417.692111613"
Oct 29 22:49:42.123576 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.123260    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podStartSLOduration=2.123243272 podStartE2EDuration="2.123243272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:42.069700511 +0000 UTC m=+417.692611533" watchObservedRunningTime="2026-10-29 22:49:42.123243272 +0000 UTC m=+417.746154293"
Oct 29 22:49:42.227266 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.226741    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:42.227723 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.227695    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:42.227856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.227743    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:42.242659 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.242268    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:42.243326 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.243303    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:42.243391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:42.243349    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:43.226672 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:43.226639    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:43.227054 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:43.226694    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:43.242124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:43.242102    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:43.242287 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:43.242270    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:44.226210 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:44.226167    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:44.226417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:44.226225    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:44.242647 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:44.242610    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:44.243031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:44.242656    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:45.226980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:45.226620    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:45.226980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:45.226668    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:45.242325 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:45.242299    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:45.242437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:45.242340    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:46.225969 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:46.225939    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:46.226404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:46.226380    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:46.242328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:46.242306    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:46.242495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:46.242478    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:47.226279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:47.226237    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:47.226797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:47.226298    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:47.242089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:47.242055    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:47.242253 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:47.242102    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:48.226052 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:48.226012    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:48.226208 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:48.226078    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:48.242465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:48.242434    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:48.242809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:48.242493    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:49.225984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:49.225948    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:49:49.226147 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:49.226015    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 29 22:49:49.242339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:49.242310    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 29 22:49:49.242437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:49.242363    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 29 22:49:50.229489 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:50.229458    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:50.229918 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:50.229712    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:50.230510 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:50.230492    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 22:49:50.246436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:50.246416    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:50.246696 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:50.246671    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:50.247278 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:50.247261    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt"
Oct 29 22:49:55.993876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:55.993836    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"]
Oct 29 22:49:55.999056 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:55.999033    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.028078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.028042    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"]
Oct 29 22:49:56.069346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069310    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069357    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069424    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069452    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069472    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3debd53f-0d24-45f7-adae-dd8331f42ebb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069632 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069588    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069632 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069617    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7b7\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-kube-api-access-8g7b7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069695 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069641    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.069695 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.069681    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170600    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170642    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170663    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170690    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3debd53f-0d24-45f7-adae-dd8331f42ebb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170720    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170744    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7b7\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-kube-api-access-8g7b7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170808    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.170854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170843    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.171222 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.170901    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.171222 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.171144    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.171222 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.171150    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.171408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.171387    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.171477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.171427    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.171510 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.171482    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3debd53f-0d24-45f7-adae-dd8331f42ebb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.173218 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.173197    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.173424 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.173408    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.193822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.193796    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.193993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.193977    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7b7\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-kube-api-access-8g7b7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnpsr\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.312109 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.312034    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:56.623687 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.623245    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"]
Oct 29 22:49:56.627739 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:49:56.627713    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3debd53f_0d24_45f7_adae_dd8331f42ebb.slice/crio-1e18dddef368a1741416fe946c58c8f500b6c7fdc6c67eea3b23686339220abd WatchSource:0}: Error finding container 1e18dddef368a1741416fe946c58c8f500b6c7fdc6c67eea3b23686339220abd: Status 404 returned error can't find the container with id 1e18dddef368a1741416fe946c58c8f500b6c7fdc6c67eea3b23686339220abd
Oct 29 22:49:56.629480 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.629446    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:56.629591 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.629525    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:56.629591 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:56.629570    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:49:57.082479 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:57.082381    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" event={"ID":"3debd53f-0d24-45f7-adae-dd8331f42ebb","Type":"ContainerStarted","Data":"8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d"}
Oct 29 22:49:57.082479 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:57.082447    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" event={"ID":"3debd53f-0d24-45f7-adae-dd8331f42ebb","Type":"ContainerStarted","Data":"1e18dddef368a1741416fe946c58c8f500b6c7fdc6c67eea3b23686339220abd"}
Oct 29 22:49:57.157327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:57.157274    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podStartSLOduration=2.157246901 podStartE2EDuration="2.157246901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:57.156198638 +0000 UTC m=+432.779109687" watchObservedRunningTime="2026-10-29 22:49:57.157246901 +0000 UTC m=+432.780157922"
Oct 29 22:49:57.312838 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:57.312809    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:49:57.314220 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:57.314195    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:49:57.314342 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:57.314241    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:49:58.313206 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:58.313171    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:49:58.313558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:58.313229    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:49:59.312821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:59.312784    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:49:59.313036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:49:59.312842    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:50:00.312959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:00.312926    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:50:00.313361 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:00.312979    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:50:01.312877 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:01.312836    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:50:01.313094 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:01.312892    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:50:02.312421 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:02.312384    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:50:02.312633 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:02.312448    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:50:02.822945 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:50:02.822908    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:50:03.313206 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:03.313173    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:50:03.313427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:03.313238    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:50:04.313407 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:04.313365    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:50:04.313792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:04.313433    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 29 22:50:05.316622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:05.316593    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:50:05.317050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:05.316892    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:50:05.317584 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:05.317569    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 22:50:20.101035 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.100999    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"]
Oct 29 22:50:20.106352 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.106325    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.123485 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.123462    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-31226c-openshift-default-dockercfg-cmhxj\""
Oct 29 22:50:20.137098 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.137074    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"]
Oct 29 22:50:20.179363 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179329    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179522 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179371    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcdk\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-kube-api-access-tlcdk\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179522 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179449    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179522 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179513    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179651 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179562    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179712 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179668    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179713    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3128da-71cb-41a0-a907-f1c381ea16fa-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179794    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.179906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.179846    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280567 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280529    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280567 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280566    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3128da-71cb-41a0-a907-f1c381ea16fa-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280596    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280641    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280672    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280698    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcdk\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-kube-api-access-tlcdk\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280722    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.280851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280793    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.281135 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.280852    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.281195 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.281127    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.281250 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.281190    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.281306 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.281285    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.281410 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.281381    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.281592 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.281576    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3128da-71cb-41a0-a907-f1c381ea16fa-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.283391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.283365    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.283717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.283699    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.313218 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.313187    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.313330 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.313285    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcdk\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-kube-api-access-tlcdk\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.419221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.419185    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:20.465894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.464595    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"]
Oct 29 22:50:20.469014 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.468441    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483216    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-podinfo\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483262    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-certs\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483340    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz76k\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-kube-api-access-bz76k\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483422    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-data\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483479    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-credential-socket\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483515    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istiod-ca-cert\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483553    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-token\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483589    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-socket\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.483730 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.483695    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-envoy\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.501484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.501440    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"]
Oct 29 22:50:20.533134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.533110    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3c314108-openshift-default-dockercfg-5jrfh\""
Oct 29 22:50:20.584197 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584167    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-envoy\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584269    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-podinfo\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584301    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-certs\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584327    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz76k\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-kube-api-access-bz76k\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584363    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-data\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584408    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-credential-socket\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584434    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istiod-ca-cert\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584466    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-token\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584510    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-socket\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.584903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.584882    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-socket\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.585163 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.585134    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-credential-socket\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.585291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.585274    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-data\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.585348 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.585294    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-certs\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.585592 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.585567    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istiod-ca-cert\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.586916 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.586896    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-envoy\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.587293 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.587253    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-podinfo\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.598168 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.598140    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"]
Oct 29 22:50:20.602302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.602285    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.617444 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.617416    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1db2136c-openshift-default-dockercfg-g725l\""
Oct 29 22:50:20.619714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.619689    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-token\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.619843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.619826    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz76k\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-kube-api-access-bz76k\") pod \"e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") " pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.630130 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.630107    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"]
Oct 29 22:50:20.685002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.684937    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-data\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.684970    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-podinfo\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.685052    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-credential-socket\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.685085    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccae810-2b20-4331-993b-8e6c173bf3cc-istiod-ca-cert\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.685107    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-token\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.685134    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4j8b\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-kube-api-access-p4j8b\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.685243    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-certs\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685396 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.685376    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-envoy\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.685456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.685419    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-socket\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.750654 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.750606    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"]
Oct 29 22:50:20.758502 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:20.757613    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3128da_71cb_41a0_a907_f1c381ea16fa.slice/crio-22f8c0ea579caa5f9ceac762fc35d991e91916cd425ff730cf0bfcb617c81f96 WatchSource:0}: Error finding container 22f8c0ea579caa5f9ceac762fc35d991e91916cd425ff730cf0bfcb617c81f96: Status 404 returned error can't find the container with id 22f8c0ea579caa5f9ceac762fc35d991e91916cd425ff730cf0bfcb617c81f96
Oct 29 22:50:20.762934 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.762890    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:20.763034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.762980    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:20.763082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.763030    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:20.785927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.785902    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-credential-socket\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.785941    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccae810-2b20-4331-993b-8e6c173bf3cc-istiod-ca-cert\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.785964    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-token\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.785987    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4j8b\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-kube-api-access-p4j8b\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786013    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-certs\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786030    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-envoy\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786047    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-socket\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786378 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786120    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-data\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786378 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786144    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-podinfo\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786378 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786267    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-credential-socket\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786545 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786497    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-certs\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786589    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-socket\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.786721 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.786705    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-data\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.787221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.787199    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccae810-2b20-4331-993b-8e6c173bf3cc-istiod-ca-cert\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.788298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.788276    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-envoy\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.788676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.788656    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-podinfo\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.805143 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.805122    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:20.825501 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.825471    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"]
Oct 29 22:50:20.829334 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.829309    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.830811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.830456    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4j8b\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-kube-api-access-p4j8b\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.830811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.830566    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-token\") pod \"e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") " pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.857879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.857837    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"]
Oct 29 22:50:20.864134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.864112    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-72818393-openshift-default-dockercfg-vqswc\""
Oct 29 22:50:20.886950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.886920    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-envoy\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.886976    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-certs\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.887043    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-socket\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.887086    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-credential-socket\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.887126    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcqq\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-kube-api-access-dbcqq\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.887152    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-token\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.887190    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-data\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.887216    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-podinfo\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.887495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.887241    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29113d99-6eaf-4046-b4d1-04318b7bdf38-istiod-ca-cert\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.917601 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.917557    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.988456    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-envoy\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.988522    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-certs\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.988597    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-socket\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.989212    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-credential-socket\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.989291    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqq\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-kube-api-access-dbcqq\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.989321    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-token\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.989365    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-data\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.989394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-podinfo\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.989893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.989434    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29113d99-6eaf-4046-b4d1-04318b7bdf38-istiod-ca-cert\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.991275 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.990284    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-credential-socket\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.991275 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.990800    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29113d99-6eaf-4046-b4d1-04318b7bdf38-istiod-ca-cert\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.991275 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.991063    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-data\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.993380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.991517    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-certs\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.993380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.991968    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-socket\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.993380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.992937    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-envoy\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:20.995697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:20.995676    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-podinfo\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:21.013804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.013760    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcqq\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-kube-api-access-dbcqq\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:21.014495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.014476    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-token\") pod \"e2e-ait-72818393-openshift-default-5f4b788dc5-86854\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") " pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:21.129058 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.128809    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"]
Oct 29 22:50:21.131359 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:21.131333    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b78bb5_b3ed_4514_96ba_167bdb074da5.slice/crio-cccfba126eee0f455595855ec44e74c98cd0125332a7144630e1597a671f6b20 WatchSource:0}: Error finding container cccfba126eee0f455595855ec44e74c98cd0125332a7144630e1597a671f6b20: Status 404 returned error can't find the container with id cccfba126eee0f455595855ec44e74c98cd0125332a7144630e1597a671f6b20
Oct 29 22:50:21.134924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.134884    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.135004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.134972    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.135051 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.135019    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.145999 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.145980    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:21.185158 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.185131    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" event={"ID":"fa3128da-71cb-41a0-a907-f1c381ea16fa","Type":"ContainerStarted","Data":"4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d"}
Oct 29 22:50:21.185559 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.185189    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" event={"ID":"fa3128da-71cb-41a0-a907-f1c381ea16fa","Type":"ContainerStarted","Data":"22f8c0ea579caa5f9ceac762fc35d991e91916cd425ff730cf0bfcb617c81f96"}
Oct 29 22:50:21.188503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.188472    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" event={"ID":"c3b78bb5-b3ed-4514-96ba-167bdb074da5","Type":"ContainerStarted","Data":"cccfba126eee0f455595855ec44e74c98cd0125332a7144630e1597a671f6b20"}
Oct 29 22:50:21.238930 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.237904    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podStartSLOduration=1.237887851 podStartE2EDuration="1.237887851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:21.236402651 +0000 UTC m=+456.859313681" watchObservedRunningTime="2026-10-29 22:50:21.237887851 +0000 UTC m=+456.860798872"
Oct 29 22:50:21.238930 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.238801    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"]
Oct 29 22:50:21.247472 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:21.247433    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccae810_2b20_4331_993b_8e6c173bf3cc.slice/crio-33d5dde1b4f0955a3cab0fe09bccf0091af979e1b8b83ca6d55d2273cef8bd75 WatchSource:0}: Error finding container 33d5dde1b4f0955a3cab0fe09bccf0091af979e1b8b83ca6d55d2273cef8bd75: Status 404 returned error can't find the container with id 33d5dde1b4f0955a3cab0fe09bccf0091af979e1b8b83ca6d55d2273cef8bd75
Oct 29 22:50:21.256997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.255584    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.256997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.255692    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.256997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.255738    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.419823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.419800    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:21.426511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.426484    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:21.459157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.457334    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"]
Oct 29 22:50:21.465465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.465429    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.465529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.465517    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:21.465570 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:21.465560    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:22.193630 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.193592    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" event={"ID":"29113d99-6eaf-4046-b4d1-04318b7bdf38","Type":"ContainerStarted","Data":"da2d3341c3e6dddf6afc26dae0bae82e3c7cee8afcd3dac3ccb93727a8832adb"}
Oct 29 22:50:22.194101 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.193664    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" event={"ID":"29113d99-6eaf-4046-b4d1-04318b7bdf38","Type":"ContainerStarted","Data":"2a57d1552fcf60b7535a5f8e72b4632b35762627c8263805c829f9a651e84227"}
Oct 29 22:50:22.195488 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.195461    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" event={"ID":"9ccae810-2b20-4331-993b-8e6c173bf3cc","Type":"ContainerStarted","Data":"a9ca609f4b79b799c35f0b30e8a236d2e0022dde0a2c6ce40fdad6beedc416f3"}
Oct 29 22:50:22.195591 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.195510    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" event={"ID":"9ccae810-2b20-4331-993b-8e6c173bf3cc","Type":"ContainerStarted","Data":"33d5dde1b4f0955a3cab0fe09bccf0091af979e1b8b83ca6d55d2273cef8bd75"}
Oct 29 22:50:22.197067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.197041    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" event={"ID":"c3b78bb5-b3ed-4514-96ba-167bdb074da5","Type":"ContainerStarted","Data":"a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a"}
Oct 29 22:50:22.197292 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.197274    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:22.198120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.198103    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:22.239595 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.239547    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podStartSLOduration=2.239533007 podStartE2EDuration="2.239533007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:22.237834231 +0000 UTC m=+457.860745251" watchObservedRunningTime="2026-10-29 22:50:22.239533007 +0000 UTC m=+457.862444027"
Oct 29 22:50:22.284041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.283997    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podStartSLOduration=2.283983583 podStartE2EDuration="2.283983583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:22.282133483 +0000 UTC m=+457.905044503" watchObservedRunningTime="2026-10-29 22:50:22.283983583 +0000 UTC m=+457.906894600"
Oct 29 22:50:22.371175 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.371126    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podStartSLOduration=2.371113259 podStartE2EDuration="2.371113259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:22.370966222 +0000 UTC m=+457.993877243" watchObservedRunningTime="2026-10-29 22:50:22.371113259 +0000 UTC m=+457.994024278"
Oct 29 22:50:22.432588 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.432557    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"]
Oct 29 22:50:22.436943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.436919    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.454518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.454496    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-08e15d-openshift-default-dockercfg-kjhd8\""
Oct 29 22:50:22.459707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.459680    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"]
Oct 29 22:50:22.504738 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.504709    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.504932 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.504742    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.504987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.504959    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.505050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.505035    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqt8\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-kube-api-access-jbqt8\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.505116 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.505097    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.505153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.505133    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.505244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.505224    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f142a2-daa3-4115-876d-111e248aea92-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.505307 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.505271    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.505367 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.505350    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f142a2-daa3-4115-876d-111e248aea92-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610322    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqt8\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-kube-api-access-jbqt8\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610385    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610413    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610461    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f142a2-daa3-4115-876d-111e248aea92-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610488    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610522    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f142a2-daa3-4115-876d-111e248aea92-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610577    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610606    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.610676    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.611589    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f142a2-daa3-4115-876d-111e248aea92-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.615242    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.617797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.615583    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.635786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.626288    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.655788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.644913    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.655788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.645208    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.669787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.661587    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqt8\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-kube-api-access-jbqt8\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.669787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.664221    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.670376 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.670348    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f142a2-daa3-4115-876d-111e248aea92-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.752348 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.752271    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:22.807069 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.806099    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:22.814990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.814963    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:22.918700 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.918666    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:22.924096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:22.924074    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:23.120045 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.119974    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"]
Oct 29 22:50:23.146542 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.146506    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:23.151486 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.151462    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:23.202091 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.202058    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:23.202573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.202114    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:23.202573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.202139    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:23.202687 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.202665    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:50:23.203079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.203056    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:23.203173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.203122    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:23.453927 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:23.453889    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f142a2_daa3_4115_876d_111e248aea92.slice/crio-cefe1b633f646988025ec0ff71575e5993acd640e8ef30988bdd36c4d814be51 WatchSource:0}: Error finding container cefe1b633f646988025ec0ff71575e5993acd640e8ef30988bdd36c4d814be51: Status 404 returned error can't find the container with id cefe1b633f646988025ec0ff71575e5993acd640e8ef30988bdd36c4d814be51
Oct 29 22:50:23.456969 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.456938    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:23.457076 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.457006    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:23.457076 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:23.457043    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:24.207058 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:24.207020    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" event={"ID":"30f142a2-daa3-4115-876d-111e248aea92","Type":"ContainerStarted","Data":"1bf5d829141487d9ac93035a4bc9f506ea01c97dd40471f4dbd239af88ba8fbe"}
Oct 29 22:50:24.207540 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:24.207082    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" event={"ID":"30f142a2-daa3-4115-876d-111e248aea92","Type":"ContainerStarted","Data":"cefe1b633f646988025ec0ff71575e5993acd640e8ef30988bdd36c4d814be51"}
Oct 29 22:50:24.278781 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:24.278723    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podStartSLOduration=2.27870946 podStartE2EDuration="2.27870946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:24.266634145 +0000 UTC m=+459.889545166" watchObservedRunningTime="2026-10-29 22:50:24.27870946 +0000 UTC m=+459.901620481"
Oct 29 22:50:24.753092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:24.753060    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:24.758468 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:24.758442    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:25.219144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:25.219108    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:25.219869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:25.219847    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:50:29.076381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.076349    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"]
Oct 29 22:50:29.079993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.079975    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.110680 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.110647    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"]
Oct 29 22:50:29.116038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.115989    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e131af-openshift-default-dockercfg-796wj\""
Oct 29 22:50:29.239873 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.239824    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.240183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.240151    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.240381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.240364    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.240538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.240516    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7jf9\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-kube-api-access-c7jf9\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.240664 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.240646    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.240903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.240885    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.241001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.240925    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.241001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.240976    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.241100 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.241001    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.341937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.341845    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.341937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.341891    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.341937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.341936    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.341959    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.341999    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342039    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342099    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342149    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7jf9\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-kube-api-access-c7jf9\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342183    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342523 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342402    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342523 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342427    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342523 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342501    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342677 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342550    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.342969 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.342949    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.344430 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.344405    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.344825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.344808    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.378374 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.378339    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.378982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.378955    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7jf9\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-kube-api-access-c7jf9\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.396224 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.396200    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:29.721107 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.721072    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"]
Oct 29 22:50:29.734964 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:29.734935    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b0d56f_4958_442c_9c39_c9d7b5a1fd96.slice/crio-b3d0b1b7c0020b854131af2fa50cb32b7273563dc931ac82647f65dd72d42772 WatchSource:0}: Error finding container b3d0b1b7c0020b854131af2fa50cb32b7273563dc931ac82647f65dd72d42772: Status 404 returned error can't find the container with id b3d0b1b7c0020b854131af2fa50cb32b7273563dc931ac82647f65dd72d42772
Oct 29 22:50:29.737235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.737201    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:29.737316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.737278    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:29.737316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:29.737307    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:30.250893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:30.250863    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" event={"ID":"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96","Type":"ContainerStarted","Data":"fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f"}
Oct 29 22:50:30.251276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:30.250911    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" event={"ID":"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96","Type":"ContainerStarted","Data":"b3d0b1b7c0020b854131af2fa50cb32b7273563dc931ac82647f65dd72d42772"}
Oct 29 22:50:30.300219 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:30.300163    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podStartSLOduration=1.30014453 podStartE2EDuration="1.30014453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:30.29883898 +0000 UTC m=+465.921749997" watchObservedRunningTime="2026-10-29 22:50:30.30014453 +0000 UTC m=+465.923055550"
Oct 29 22:50:30.397073 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:30.397042    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:30.402410 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:30.402384    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:31.256492 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:31.256462    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:31.257549 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:31.257529    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:50:32.759503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.759467    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"]
Oct 29 22:50:32.763931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.763907    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:32.789743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.789718    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-08e15d\""
Oct 29 22:50:32.829049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.829018    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-08e15d-dockercfg-g6mzq\""
Oct 29 22:50:32.834465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.834364    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscg6\" (UniqueName: \"kubernetes.io/projected/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-kube-api-access-sscg6\") pod \"payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:32.839952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.839918    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:32.846595 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.846555    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"]
Oct 29 22:50:32.895758 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:50:32.895713    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:50:32.964258 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.960062    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscg6\" (UniqueName: \"kubernetes.io/projected/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-kube-api-access-sscg6\") pod \"payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:32.964258 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.960114    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:32.964258 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.961063    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:32.985425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:32.985392    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscg6\" (UniqueName: \"kubernetes.io/projected/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-kube-api-access-sscg6\") pod \"payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:33.149546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:33.148932    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:33.445802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:33.445712    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"]
Oct 29 22:50:33.449508 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:33.449476    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc1f75d_3bef_4805_9e22_b48a8d0c1099.slice/crio-8c51c5065df6541fee5b574f98d60a214ae36bf2c8620ce9c11e1ad075613037 WatchSource:0}: Error finding container 8c51c5065df6541fee5b574f98d60a214ae36bf2c8620ce9c11e1ad075613037: Status 404 returned error can't find the container with id 8c51c5065df6541fee5b574f98d60a214ae36bf2c8620ce9c11e1ad075613037
Oct 29 22:50:34.290740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:34.290697    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs" event={"ID":"9cc1f75d-3bef-4805-9e22-b48a8d0c1099","Type":"ContainerStarted","Data":"f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2"}
Oct 29 22:50:34.291197 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:34.290774    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs" event={"ID":"9cc1f75d-3bef-4805-9e22-b48a8d0c1099","Type":"ContainerStarted","Data":"8c51c5065df6541fee5b574f98d60a214ae36bf2c8620ce9c11e1ad075613037"}
Oct 29 22:50:34.291197 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:34.290874    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:34.332454 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:34.332387    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs" podStartSLOduration=2.332369253 podStartE2EDuration="2.332369253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:34.330936646 +0000 UTC m=+469.953847666" watchObservedRunningTime="2026-10-29 22:50:34.332369253 +0000 UTC m=+469.955280274"
Oct 29 22:50:37.893609 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.893526    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"]
Oct 29 22:50:37.897697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.897672    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:37.924401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.924380    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"]
Oct 29 22:50:37.933469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.933444    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e131af\""
Oct 29 22:50:37.933615 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.933533    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e131af-dockercfg-6qk7q\""
Oct 29 22:50:37.933938 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.933914    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxx22\" (UniqueName: \"kubernetes.io/projected/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-kube-api-access-cxx22\") pod \"payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:37.934037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.934019    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:37.939096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.939076    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"]
Oct 29 22:50:37.943101 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.943074    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:37.995794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:37.995702    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"]
Oct 29 22:50:38.035291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.035249    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxx22\" (UniqueName: \"kubernetes.io/projected/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-kube-api-access-cxx22\") pod \"payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:38.035431 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.035324    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7316e-15d7-41d9-9496-1d59d982b4d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:38.035431 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.035399    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:38.035518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.035432    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8k5q\" (UniqueName: \"kubernetes.io/projected/4ab7316e-15d7-41d9-9496-1d59d982b4d9-kube-api-access-j8k5q\") pod \"payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:38.036356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.036337    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:38.060557 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.060527    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxx22\" (UniqueName: \"kubernetes.io/projected/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-kube-api-access-cxx22\") pod \"payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:38.136328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.136297    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8k5q\" (UniqueName: \"kubernetes.io/projected/4ab7316e-15d7-41d9-9496-1d59d982b4d9-kube-api-access-j8k5q\") pod \"payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:38.136489 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.136399    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7316e-15d7-41d9-9496-1d59d982b4d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:38.137060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.137040    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7316e-15d7-41d9-9496-1d59d982b4d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:38.159294 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.159231    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8k5q\" (UniqueName: \"kubernetes.io/projected/4ab7316e-15d7-41d9-9496-1d59d982b4d9-kube-api-access-j8k5q\") pod \"payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:38.211957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.211928    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:38.269223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.268956    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:38.642296 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.642264    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"]
Oct 29 22:50:38.656060 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:38.656028    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90a2c0c_652b_4f6b_9f19_efff8dff39fb.slice/crio-417817a51786e93a3116a11b63b49c120f6da0ce3bb8f4e723dc78e46fdd5ade WatchSource:0}: Error finding container 417817a51786e93a3116a11b63b49c120f6da0ce3bb8f4e723dc78e46fdd5ade: Status 404 returned error can't find the container with id 417817a51786e93a3116a11b63b49c120f6da0ce3bb8f4e723dc78e46fdd5ade
Oct 29 22:50:38.703658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:38.703636    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"]
Oct 29 22:50:38.709865 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:38.709836    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab7316e_15d7_41d9_9496_1d59d982b4d9.slice/crio-118956687d172f729913ae7bb777effd85b6ed96589c9f836eba45de5640b587 WatchSource:0}: Error finding container 118956687d172f729913ae7bb777effd85b6ed96589c9f836eba45de5640b587: Status 404 returned error can't find the container with id 118956687d172f729913ae7bb777effd85b6ed96589c9f836eba45de5640b587
Oct 29 22:50:39.423004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.422722    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" event={"ID":"4ab7316e-15d7-41d9-9496-1d59d982b4d9","Type":"ContainerStarted","Data":"4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0"}
Oct 29 22:50:39.423004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.422849    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:39.423004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.422875    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" event={"ID":"4ab7316e-15d7-41d9-9496-1d59d982b4d9","Type":"ContainerStarted","Data":"118956687d172f729913ae7bb777effd85b6ed96589c9f836eba45de5640b587"}
Oct 29 22:50:39.424653 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.424621    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" event={"ID":"a90a2c0c-652b-4f6b-9f19-efff8dff39fb","Type":"ContainerStarted","Data":"d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813"}
Oct 29 22:50:39.424821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.424662    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" event={"ID":"a90a2c0c-652b-4f6b-9f19-efff8dff39fb","Type":"ContainerStarted","Data":"417817a51786e93a3116a11b63b49c120f6da0ce3bb8f4e723dc78e46fdd5ade"}
Oct 29 22:50:39.424821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.424700    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:39.552423 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.552277    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" podStartSLOduration=2.552261351 podStartE2EDuration="2.552261351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:39.511915345 +0000 UTC m=+475.134826387" watchObservedRunningTime="2026-10-29 22:50:39.552261351 +0000 UTC m=+475.175172372"
Oct 29 22:50:39.553288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.553238    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" podStartSLOduration=2.553222736 podStartE2EDuration="2.553222736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:39.550943641 +0000 UTC m=+475.173854662" watchObservedRunningTime="2026-10-29 22:50:39.553222736 +0000 UTC m=+475.176133757"
Oct 29 22:50:39.603629 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.603577    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"]
Oct 29 22:50:39.617291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.617261    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.624882 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.624842    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"]
Oct 29 22:50:39.632914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.632722    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:50:39.632914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.632844    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:50:39.720784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.720345    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.720784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.720471    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.720784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.720507    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt2rr\" (UniqueName: \"kubernetes.io/projected/77d6dc6c-124b-47e5-a04a-0adb3a87d113-kube-api-access-rt2rr\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.822880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.821644    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.822880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.821708    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt2rr\" (UniqueName: \"kubernetes.io/projected/77d6dc6c-124b-47e5-a04a-0adb3a87d113-kube-api-access-rt2rr\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.822880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.821823    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.822880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.822692    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.826340 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.826315    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.846750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.846727    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt2rr\" (UniqueName: \"kubernetes.io/projected/77d6dc6c-124b-47e5-a04a-0adb3a87d113-kube-api-access-rt2rr\") pod \"maas-api-revoke-keys-e2e-migrate-639067f9-8btnk\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:39.939202 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:39.939168    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:40.399850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:40.398032    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"]
Oct 29 22:50:40.421114 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:40.421092    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:50:40.430331 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:40.430303    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk" event={"ID":"77d6dc6c-124b-47e5-a04a-0adb3a87d113","Type":"ContainerStarted","Data":"150b0165d0848879f521559194fcf292851c3e67af41ce2b11b943b6dbdec07b"}
Oct 29 22:50:45.298855 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:45.298246    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 22:50:47.526206 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.526124    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk" event={"ID":"77d6dc6c-124b-47e5-a04a-0adb3a87d113","Type":"ContainerStarted","Data":"6cad1252710efef7973e047b9f0269245afb03713aaab88082d42730eda4352e"}
Oct 29 22:50:47.527897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.527865    4827 generic.go:356] "Generic (PLEG): container finished" podID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerID="4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d" exitCode=0
Oct 29 22:50:47.528035 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.527926    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" event={"ID":"fa3128da-71cb-41a0-a907-f1c381ea16fa","Type":"ContainerDied","Data":"4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d"}
Oct 29 22:50:47.528323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.528305    4827 scope.go:122] "RemoveContainer" containerID="4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d"
Oct 29 22:50:47.531327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.530644    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:47.531327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.530925    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:47.531327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.530990    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:47.643737 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:47.643689    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk" podStartSLOduration=2.365107003 podStartE2EDuration="8.643677123s" totalImagesPullingTime="6.27857012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:47.596715744 +0000 UTC m=+483.219626774" watchObservedRunningTime="2026-10-29 22:50:47.643677123 +0000 UTC m=+483.266588191"
Oct 29 22:50:48.535405 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:48.535365    4827 generic.go:356] "Generic (PLEG): container finished" podID="77d6dc6c-124b-47e5-a04a-0adb3a87d113" containerID="6cad1252710efef7973e047b9f0269245afb03713aaab88082d42730eda4352e" exitCode=0
Oct 29 22:50:48.535860 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:48.535446    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk" event={"ID":"77d6dc6c-124b-47e5-a04a-0adb3a87d113","Type":"ContainerDied","Data":"6cad1252710efef7973e047b9f0269245afb03713aaab88082d42730eda4352e"}
Oct 29 22:50:48.537571 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:48.537547    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" event={"ID":"fa3128da-71cb-41a0-a907-f1c381ea16fa","Type":"ContainerStarted","Data":"121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45"}
Oct 29 22:50:49.426788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.425872    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:49.426788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.426033    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:50:49.426788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.426073    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:50:49.836431 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.836124    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:49.907324 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.906925    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-openshift-ca\") pod \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") "
Oct 29 22:50:49.907324 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.906972    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d6dc6c-124b-47e5-a04a-0adb3a87d113-kube-api-access-rt2rr\" (UniqueName: \"kubernetes.io/projected/77d6dc6c-124b-47e5-a04a-0adb3a87d113-kube-api-access-rt2rr\") pod \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") "
Oct 29 22:50:49.907324 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.907017    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-xks-ca\") pod \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\" (UID: \"77d6dc6c-124b-47e5-a04a-0adb3a87d113\") "
Oct 29 22:50:49.907601 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.907355    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-openshift-ca" pod "77d6dc6c-124b-47e5-a04a-0adb3a87d113" (UID: "77d6dc6c-124b-47e5-a04a-0adb3a87d113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:49.910043 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.910017    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d6dc6c-124b-47e5-a04a-0adb3a87d113-kube-api-access-rt2rr" pod "77d6dc6c-124b-47e5-a04a-0adb3a87d113" (UID: "77d6dc6c-124b-47e5-a04a-0adb3a87d113"). InnerVolumeSpecName "kube-api-access-rt2rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:49.911904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:49.911876    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-xks-ca" pod "77d6dc6c-124b-47e5-a04a-0adb3a87d113" (UID: "77d6dc6c-124b-47e5-a04a-0adb3a87d113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:50:50.007962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.007878    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:50:50.007962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.007909    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt2rr\" (UniqueName: \"kubernetes.io/projected/77d6dc6c-124b-47e5-a04a-0adb3a87d113-kube-api-access-rt2rr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:50:50.007962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.007919    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d6dc6c-124b-47e5-a04a-0adb3a87d113-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:50:50.420619 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.420578    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:50:50.420844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.420641    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:50:50.433565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.433539    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 22:50:50.433715 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.433612    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 22:50:50.506927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.506898    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"]
Oct 29 22:50:50.507491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.507468    4827 cpu_manager.go:414] "containerMap: removing container" podUID="77d6dc6c-124b-47e5-a04a-0adb3a87d113" containerName="revoke-keys"
Oct 29 22:50:50.507491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.507493    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d6dc6c-124b-47e5-a04a-0adb3a87d113" containerName="revoke-keys"
Oct 29 22:50:50.507690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.507629    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d6dc6c-124b-47e5-a04a-0adb3a87d113" containerName="revoke-keys"
Oct 29 22:50:50.513994 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.513967    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"]
Oct 29 22:50:50.514125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.514081    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.558403 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.558348    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk" event={"ID":"77d6dc6c-124b-47e5-a04a-0adb3a87d113","Type":"ContainerDied","Data":"150b0165d0848879f521559194fcf292851c3e67af41ce2b11b943b6dbdec07b"}
Oct 29 22:50:50.558579 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.558452    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="150b0165d0848879f521559194fcf292851c3e67af41ce2b11b943b6dbdec07b"
Oct 29 22:50:50.558579 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.558537    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"
Oct 29 22:50:50.571789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.562794    4827 generic.go:356] "Generic (PLEG): container finished" podID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerID="a9ca609f4b79b799c35f0b30e8a236d2e0022dde0a2c6ce40fdad6beedc416f3" exitCode=0
Oct 29 22:50:50.580796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.562842    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" event={"ID":"9ccae810-2b20-4331-993b-8e6c173bf3cc","Type":"ContainerDied","Data":"a9ca609f4b79b799c35f0b30e8a236d2e0022dde0a2c6ce40fdad6beedc416f3"}
Oct 29 22:50:50.580796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.579465    4827 scope.go:122] "RemoveContainer" containerID="a9ca609f4b79b799c35f0b30e8a236d2e0022dde0a2c6ce40fdad6beedc416f3"
Oct 29 22:50:50.598425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.597548    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:50.598425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.597692    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:50.598425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.597854    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:50.599153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.598795    4827 generic.go:356] "Generic (PLEG): container finished" podID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerID="da2d3341c3e6dddf6afc26dae0bae82e3c7cee8afcd3dac3ccb93727a8832adb" exitCode=0
Oct 29 22:50:50.599153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.598945    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" event={"ID":"29113d99-6eaf-4046-b4d1-04318b7bdf38","Type":"ContainerDied","Data":"da2d3341c3e6dddf6afc26dae0bae82e3c7cee8afcd3dac3ccb93727a8832adb"}
Oct 29 22:50:50.599450 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.599433    4827 scope.go:122] "RemoveContainer" containerID="da2d3341c3e6dddf6afc26dae0bae82e3c7cee8afcd3dac3ccb93727a8832adb"
Oct 29 22:50:50.605041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.601795    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:50.605041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.601885    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:50.605041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.601930    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:50:50.616437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.616410    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.616532 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.616452    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.616702 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.616681    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7lg\" (UniqueName: \"kubernetes.io/projected/6b4eb15e-ac9a-47f5-8f31-761eee10d265-kube-api-access-5q7lg\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.719592 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.719539    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7lg\" (UniqueName: \"kubernetes.io/projected/6b4eb15e-ac9a-47f5-8f31-761eee10d265-kube-api-access-5q7lg\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.719760 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.719610    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.719760 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.719646    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.722583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.721122    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.733787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.730924    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.747546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.747492    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7lg\" (UniqueName: \"kubernetes.io/projected/6b4eb15e-ac9a-47f5-8f31-761eee10d265-kube-api-access-5q7lg\") pod \"maas-api-revoke-keys-e2e-ait-72818393-s5h9s\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:50.833788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:50.833287    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:51.284841 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.281061    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"]
Oct 29 22:50:51.329281 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:50:51.325084    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4eb15e_ac9a_47f5_8f31_761eee10d265.slice/crio-79d0010903fa9f61cc678c94b4239e8fe0c4b05b5b6cbb7be637cb9f000824ae WatchSource:0}: Error finding container 79d0010903fa9f61cc678c94b4239e8fe0c4b05b5b6cbb7be637cb9f000824ae: Status 404 returned error can't find the container with id 79d0010903fa9f61cc678c94b4239e8fe0c4b05b5b6cbb7be637cb9f000824ae
Oct 29 22:50:51.420282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.420239    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:50:51.420519 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.420493    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:50:51.641816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.629405    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s" event={"ID":"6b4eb15e-ac9a-47f5-8f31-761eee10d265","Type":"ContainerStarted","Data":"79d0010903fa9f61cc678c94b4239e8fe0c4b05b5b6cbb7be637cb9f000824ae"}
Oct 29 22:50:51.641816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.632186    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" event={"ID":"29113d99-6eaf-4046-b4d1-04318b7bdf38","Type":"ContainerStarted","Data":"55550a5f6051bf7709660fda246d561a3a5ba264dc2a603bc25cfc83fa100bd1"}
Oct 29 22:50:51.641816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.635657    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" event={"ID":"9ccae810-2b20-4331-993b-8e6c173bf3cc","Type":"ContainerStarted","Data":"06be3f0b35829355a9d8509fcc4dad9a125c9e245fc4145430705514f732cced"}
Oct 29 22:50:51.920847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.920633    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:51.967805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.951151    4827 patch_prober.go:29] interesting pod/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:50:51.967805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:51.951208    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:50:52.147320 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.147224    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:52.147960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.147608    4827 patch_prober.go:29] interesting pod/e2e-ait-72818393-openshift-default-5f4b788dc5-86854 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:50:52.147960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.147667    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:50:52.216827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.211873    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:52.450795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.450259    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:50:52.450795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.450342    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:50:52.750398 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.748841    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s" event={"ID":"6b4eb15e-ac9a-47f5-8f31-761eee10d265","Type":"ContainerStarted","Data":"3c98a22a12aa71c7ccc9720fa411f5ef4abf8b4795802033a110eac86ba88e85"}
Oct 29 22:50:52.919791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.919193    4827 patch_prober.go:29] interesting pod/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:50:52.919791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:52.919266    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:50:53.147789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.146726    4827 patch_prober.go:29] interesting pod/e2e-ait-72818393-openshift-default-5f4b788dc5-86854 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:50:53.163790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.161205    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:50:53.203871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.202149    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:53.203871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.202247    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:53.461848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.423179    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:50:53.461848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.423242    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:50:53.803013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.801254    4827 generic.go:356] "Generic (PLEG): container finished" podID="6b4eb15e-ac9a-47f5-8f31-761eee10d265" containerID="3c98a22a12aa71c7ccc9720fa411f5ef4abf8b4795802033a110eac86ba88e85" exitCode=0
Oct 29 22:50:53.803013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.802317    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s" event={"ID":"6b4eb15e-ac9a-47f5-8f31-761eee10d265","Type":"ContainerDied","Data":"3c98a22a12aa71c7ccc9720fa411f5ef4abf8b4795802033a110eac86ba88e85"}
Oct 29 22:50:53.958798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.929852    4827 patch_prober.go:29] interesting pod/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:50:53.958798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:53.929918    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:50:54.176723 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.176239    4827 patch_prober.go:29] interesting pod/e2e-ait-72818393-openshift-default-5f4b788dc5-86854 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:50:54.176723 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.176301    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:50:54.346613 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.346579    4827 patch_prober.go:29] interesting pod/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:50:54.346894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.346866    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:50:54.422474 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.422129    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:50:54.422474 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.422191    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:50:54.933984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.933949    4827 patch_prober.go:29] interesting pod/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:50:54.972608 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:54.972568    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:50:55.177692 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.177655    4827 patch_prober.go:29] interesting pod/e2e-ait-72818393-openshift-default-5f4b788dc5-86854 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:50:55.177992 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.177961    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:50:55.347104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.347000    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"]
Oct 29 22:50:55.382530 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.374206    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.382530 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.382453    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"]
Oct 29 22:50:55.631791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.630470    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.631791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.630625    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.631791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.630663    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hk29\" (UniqueName: \"kubernetes.io/projected/05ff09f2-e001-43ed-9b90-8b5404432df4-kube-api-access-7hk29\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.808825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.749355    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.808825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.749410    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hk29\" (UniqueName: \"kubernetes.io/projected/05ff09f2-e001-43ed-9b90-8b5404432df4-kube-api-access-7hk29\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.808825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.749522    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.833815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.809576    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:55.943838 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.932964    4827 patch_prober.go:29] interesting pod/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:50:55.943838 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.933053    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:50:55.997797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:55.993233    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hk29\" (UniqueName: \"kubernetes.io/projected/05ff09f2-e001-43ed-9b90-8b5404432df4-kube-api-access-7hk29\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:56.031213 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.030690    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:56.152792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.150061    4827 patch_prober.go:29] interesting pod/e2e-ait-72818393-openshift-default-5f4b788dc5-86854 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:50:56.152792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.150122    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:50:56.152792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.151691    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:56.213842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.209247    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:50:56.350788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.346167    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:56.367786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.365079    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 22:50:56.396800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.396592    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4eb15e-ac9a-47f5-8f31-761eee10d265-kube-api-access-5q7lg\" (UniqueName: \"kubernetes.io/projected/6b4eb15e-ac9a-47f5-8f31-761eee10d265-kube-api-access-5q7lg\") pod \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") "
Oct 29 22:50:56.396800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.396668    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-xks-ca\") pod \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") "
Oct 29 22:50:56.397062 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.397014    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-openshift-ca\") pod \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\" (UID: \"6b4eb15e-ac9a-47f5-8f31-761eee10d265\") "
Oct 29 22:50:56.400784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.398452    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-openshift-ca" pod "6b4eb15e-ac9a-47f5-8f31-761eee10d265" (UID: "6b4eb15e-ac9a-47f5-8f31-761eee10d265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:56.460789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.459936    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-xks-ca" pod "6b4eb15e-ac9a-47f5-8f31-761eee10d265" (UID: "6b4eb15e-ac9a-47f5-8f31-761eee10d265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:50:56.481846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.467908    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4eb15e-ac9a-47f5-8f31-761eee10d265-kube-api-access-5q7lg" pod "6b4eb15e-ac9a-47f5-8f31-761eee10d265" (UID: "6b4eb15e-ac9a-47f5-8f31-761eee10d265"). InnerVolumeSpecName "kube-api-access-5q7lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:56.503853 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.501537    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7lg\" (UniqueName: \"kubernetes.io/projected/6b4eb15e-ac9a-47f5-8f31-761eee10d265-kube-api-access-5q7lg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:50:56.503853 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.501578    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:50:56.503853 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:56.501595    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb15e-ac9a-47f5-8f31-761eee10d265-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:50:57.039787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.025564    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"]
Oct 29 22:50:57.110794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.110651    4827 patch_prober.go:29] interesting pod/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:50:57.110794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.110697    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:50:57.175810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.168386    4827 patch_prober.go:29] interesting pod/e2e-ait-72818393-openshift-default-5f4b788dc5-86854 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:50:57.175810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.168445    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:50:57.200079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.196300    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb" event={"ID":"05ff09f2-e001-43ed-9b90-8b5404432df4","Type":"ContainerStarted","Data":"a9a4168d7e472512cb6ef779df6e68f990a988b0de169fceb1fb0f0791b8b112"}
Oct 29 22:50:57.240119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.240077    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"
Oct 29 22:50:57.240967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.240942    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s" event={"ID":"6b4eb15e-ac9a-47f5-8f31-761eee10d265","Type":"ContainerDied","Data":"79d0010903fa9f61cc678c94b4239e8fe0c4b05b5b6cbb7be637cb9f000824ae"}
Oct 29 22:50:57.241144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.241126    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d0010903fa9f61cc678c94b4239e8fe0c4b05b5b6cbb7be637cb9f000824ae"
Oct 29 22:50:57.921272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.920830    4827 patch_prober.go:29] interesting pod/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:50:57.921272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:57.920885    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:50:58.159797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:58.154137    4827 patch_prober.go:29] interesting pod/e2e-ait-72818393-openshift-default-5f4b788dc5-86854 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:50:58.160413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:58.160377    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:50:58.314788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:58.306914    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb" event={"ID":"05ff09f2-e001-43ed-9b90-8b5404432df4","Type":"ContainerStarted","Data":"8bc2319d713ab28d969040c67e764544ff35f0518bf9b95a26678ce88316d5a5"}
Oct 29 22:50:59.051008 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.050973    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:59.052851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.052818    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:50:59.127603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.127080    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb" podStartSLOduration=4.127061869 podStartE2EDuration="4.127061869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:58.363568967 +0000 UTC m=+493.986479989" watchObservedRunningTime="2026-10-29 22:50:59.127061869 +0000 UTC m=+494.749972889"
Oct 29 22:50:59.152649 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.152621    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:59.154038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.154017    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:50:59.313982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.313903    4827 generic.go:356] "Generic (PLEG): container finished" podID="05ff09f2-e001-43ed-9b90-8b5404432df4" containerID="8bc2319d713ab28d969040c67e764544ff35f0518bf9b95a26678ce88316d5a5" exitCode=0
Oct 29 22:50:59.314373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.314008    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb" event={"ID":"05ff09f2-e001-43ed-9b90-8b5404432df4","Type":"ContainerDied","Data":"8bc2319d713ab28d969040c67e764544ff35f0518bf9b95a26678ce88316d5a5"}
Oct 29 22:50:59.791686 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.791647    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"]
Oct 29 22:50:59.792614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.792580    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6b4eb15e-ac9a-47f5-8f31-761eee10d265" containerName="revoke-keys"
Oct 29 22:50:59.792802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.792786    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4eb15e-ac9a-47f5-8f31-761eee10d265" containerName="revoke-keys"
Oct 29 22:50:59.793208 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.793180    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4eb15e-ac9a-47f5-8f31-761eee10d265" containerName="revoke-keys"
Oct 29 22:50:59.802684 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.802665    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:50:59.813527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.813501    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"]
Oct 29 22:50:59.887509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.887464    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:50:59.887867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.887840    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hkw\" (UniqueName: \"kubernetes.io/projected/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-kube-api-access-c4hkw\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:50:59.888013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.887996    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:50:59.992449 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.992407    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:50:59.992711 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.992693    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hkw\" (UniqueName: \"kubernetes.io/projected/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-kube-api-access-c4hkw\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:50:59.992867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.992851    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:50:59.993816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:50:59.993791    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:51:00.009681 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.009644    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:51:00.018815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.018789    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hkw\" (UniqueName: \"kubernetes.io/projected/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-kube-api-access-c4hkw\") pod \"maas-api-revoke-keys-e2e-derive-3c314108-jjpwt\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:51:00.119372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.119291    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:51:00.467616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.467435    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"]
Oct 29 22:51:00.532029 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:00.531989    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b4a8a3_222e_48d1_bb6a_36ff3aed6bca.slice/crio-6b0f88f1a91e338034e3864c6f7be4b0cb2127deb780bdf39e0158e42539845a WatchSource:0}: Error finding container 6b0f88f1a91e338034e3864c6f7be4b0cb2127deb780bdf39e0158e42539845a: Status 404 returned error can't find the container with id 6b0f88f1a91e338034e3864c6f7be4b0cb2127deb780bdf39e0158e42539845a
Oct 29 22:51:00.626167 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.626131    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:51:00.733937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.733911    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ff09f2-e001-43ed-9b90-8b5404432df4-kube-api-access-7hk29\" (UniqueName: \"kubernetes.io/projected/05ff09f2-e001-43ed-9b90-8b5404432df4-kube-api-access-7hk29\") pod \"05ff09f2-e001-43ed-9b90-8b5404432df4\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") "
Oct 29 22:51:00.734117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.733989    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-openshift-ca\") pod \"05ff09f2-e001-43ed-9b90-8b5404432df4\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") "
Oct 29 22:51:00.734117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.734025    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-xks-ca\") pod \"05ff09f2-e001-43ed-9b90-8b5404432df4\" (UID: \"05ff09f2-e001-43ed-9b90-8b5404432df4\") "
Oct 29 22:51:00.734342 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.734318    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-openshift-ca" pod "05ff09f2-e001-43ed-9b90-8b5404432df4" (UID: "05ff09f2-e001-43ed-9b90-8b5404432df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:51:00.734420 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.734400    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:00.736775 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.736732    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-xks-ca" pod "05ff09f2-e001-43ed-9b90-8b5404432df4" (UID: "05ff09f2-e001-43ed-9b90-8b5404432df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:51:00.736865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.736735    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ff09f2-e001-43ed-9b90-8b5404432df4-kube-api-access-7hk29" pod "05ff09f2-e001-43ed-9b90-8b5404432df4" (UID: "05ff09f2-e001-43ed-9b90-8b5404432df4"). InnerVolumeSpecName "kube-api-access-7hk29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:00.835123 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.835089    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ff09f2-e001-43ed-9b90-8b5404432df4-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:00.835123 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:00.835118    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hk29\" (UniqueName: \"kubernetes.io/projected/05ff09f2-e001-43ed-9b90-8b5404432df4-kube-api-access-7hk29\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:01.329030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:01.328893    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt" event={"ID":"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca","Type":"ContainerStarted","Data":"16617e257da7ac10a1e67c682d564fd1c16db19a363bf389f959b0b5294ef3d0"}
Oct 29 22:51:01.329030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:01.328967    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt" event={"ID":"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca","Type":"ContainerStarted","Data":"6b0f88f1a91e338034e3864c6f7be4b0cb2127deb780bdf39e0158e42539845a"}
Oct 29 22:51:01.330385 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:01.330360    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb" event={"ID":"05ff09f2-e001-43ed-9b90-8b5404432df4","Type":"ContainerDied","Data":"a9a4168d7e472512cb6ef779df6e68f990a988b0de169fceb1fb0f0791b8b112"}
Oct 29 22:51:01.330518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:01.330394    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"
Oct 29 22:51:01.330518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:01.330404    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9a4168d7e472512cb6ef779df6e68f990a988b0de169fceb1fb0f0791b8b112"
Oct 29 22:51:01.398066 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:01.398003    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt" podStartSLOduration=2.397979801 podStartE2EDuration="2.397979801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:01.372380553 +0000 UTC m=+496.995291575" watchObservedRunningTime="2026-10-29 22:51:01.397979801 +0000 UTC m=+497.020890821"
Oct 29 22:51:02.257679 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.257641    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:51:02.258188 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.257706    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:51:02.338531 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.338496    4827 generic.go:356] "Generic (PLEG): container finished" podID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerID="a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a" exitCode=0
Oct 29 22:51:02.338712 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.338573    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" event={"ID":"c3b78bb5-b3ed-4514-96ba-167bdb074da5","Type":"ContainerDied","Data":"a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a"}
Oct 29 22:51:02.339149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.339069    4827 scope.go:122] "RemoveContainer" containerID="a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a"
Oct 29 22:51:02.341539 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.341507    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:02.341646 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.341593    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:02.341709 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.341643    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:02.342388 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.342363    4827 generic.go:356] "Generic (PLEG): container finished" podID="d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" containerID="16617e257da7ac10a1e67c682d564fd1c16db19a363bf389f959b0b5294ef3d0" exitCode=0
Oct 29 22:51:02.342487 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.342401    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt" event={"ID":"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca","Type":"ContainerDied","Data":"16617e257da7ac10a1e67c682d564fd1c16db19a363bf389f959b0b5294ef3d0"}
Oct 29 22:51:02.803790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.800212    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"]
Oct 29 22:51:02.821226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:02.819622    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72818393-s5h9s"]
Oct 29 22:51:02.883318 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:51:02.883268    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:51:03.350606 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.350572    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" event={"ID":"c3b78bb5-b3ed-4514-96ba-167bdb074da5","Type":"ContainerStarted","Data":"5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd"}
Oct 29 22:51:03.354209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.354187    4827 generic.go:356] "Generic (PLEG): container finished" podID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerID="fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f" exitCode=0
Oct 29 22:51:03.354325 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.354259    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" event={"ID":"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96","Type":"ContainerDied","Data":"fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f"}
Oct 29 22:51:03.355362 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.354703    4827 scope.go:122] "RemoveContainer" containerID="fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f"
Oct 29 22:51:03.355982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.355949    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:03.356064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.356025    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:03.356109 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.356061    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:03.619280 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.618981    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:51:03.713451 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.713410    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-openshift-ca\") pod \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") "
Oct 29 22:51:03.713620 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.713479    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-kube-api-access-c4hkw\" (UniqueName: \"kubernetes.io/projected/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-kube-api-access-c4hkw\") pod \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") "
Oct 29 22:51:03.713666 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.713620    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-xks-ca\") pod \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\" (UID: \"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca\") "
Oct 29 22:51:03.713881 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.713844    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-openshift-ca" pod "d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" (UID: "d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:51:03.714033 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.714012    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:03.716018 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.715990    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-xks-ca" pod "d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" (UID: "d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:51:03.717007 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.716985    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-kube-api-access-c4hkw" pod "d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" (UID: "d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca"). InnerVolumeSpecName "kube-api-access-c4hkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:03.813788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.811105    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:51:03.819789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.815367    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:03.819789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.815398    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hkw\" (UniqueName: \"kubernetes.io/projected/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca-kube-api-access-c4hkw\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:03.843790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.823924    4827 patch_prober.go:29] interesting pod/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:03.843790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:03.823981    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:04.337827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.337796    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"]
Oct 29 22:51:04.338026 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.338010    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy" containerID="cri-o://55550a5f6051bf7709660fda246d561a3a5ba264dc2a603bc25cfc83fa100bd1" gracePeriod=30
Oct 29 22:51:04.398802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.388384    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt" event={"ID":"d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca","Type":"ContainerDied","Data":"6b0f88f1a91e338034e3864c6f7be4b0cb2127deb780bdf39e0158e42539845a"}
Oct 29 22:51:04.398802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.388457    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0f88f1a91e338034e3864c6f7be4b0cb2127deb780bdf39e0158e42539845a"
Oct 29 22:51:04.398802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.388559    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"
Oct 29 22:51:04.407867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.406597    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" event={"ID":"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96","Type":"ContainerStarted","Data":"a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5"}
Oct 29 22:51:04.568795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.562967    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4eb15e-ac9a-47f5-8f31-761eee10d265" path="/var/lib/kubelet/pods/6b4eb15e-ac9a-47f5-8f31-761eee10d265/volumes"
Oct 29 22:51:04.812788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.812225    4827 patch_prober.go:29] interesting pod/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:04.812788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:04.812290    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:05.405863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.401218    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:51:05.405863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.401388    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:51:05.405863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.401475    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:51:05.487473 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.478047    4827 generic.go:356] "Generic (PLEG): container finished" podID="30f142a2-daa3-4115-876d-111e248aea92" containerID="1bf5d829141487d9ac93035a4bc9f506ea01c97dd40471f4dbd239af88ba8fbe" exitCode=0
Oct 29 22:51:05.487473 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.478992    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" event={"ID":"30f142a2-daa3-4115-876d-111e248aea92","Type":"ContainerDied","Data":"1bf5d829141487d9ac93035a4bc9f506ea01c97dd40471f4dbd239af88ba8fbe"}
Oct 29 22:51:05.487473 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.479355    4827 scope.go:122] "RemoveContainer" containerID="1bf5d829141487d9ac93035a4bc9f506ea01c97dd40471f4dbd239af88ba8fbe"
Oct 29 22:51:05.500758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.499858    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:05.500758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.499968    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:05.500758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.500045    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:05.734787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.734708    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"]
Oct 29 22:51:05.753617 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.753468    4827 cpu_manager.go:414] "containerMap: removing container" podUID="05ff09f2-e001-43ed-9b90-8b5404432df4" containerName="revoke-keys"
Oct 29 22:51:05.755919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.754816    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ff09f2-e001-43ed-9b90-8b5404432df4" containerName="revoke-keys"
Oct 29 22:51:05.755919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.754866    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" containerName="revoke-keys"
Oct 29 22:51:05.755919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.754872    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" containerName="revoke-keys"
Oct 29 22:51:05.755919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.754960    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" containerName="revoke-keys"
Oct 29 22:51:05.755919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.754972    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ff09f2-e001-43ed-9b90-8b5404432df4" containerName="revoke-keys"
Oct 29 22:51:05.757145 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.757120    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.776719    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"]
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.789003    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-063b46-openshift-default-dockercfg-hj7tc\""
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.808291    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.822985    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d3629c-1694-4cb8-86ad-46803f41dfc0-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823022    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823074    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823119    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823161    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7kt\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-kube-api-access-5p7kt\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823187    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823287    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823413    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.808829    4827 patch_prober.go:29] interesting pod/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:05.824071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.823551    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936081    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936141    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d3629c-1694-4cb8-86ad-46803f41dfc0-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936169    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936204    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936236    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936269    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7kt\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-kube-api-access-5p7kt\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936298    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936342    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.937787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.936421    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.955569 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.955528    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.956254 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.956231    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.961593 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.958748    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.962169 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.962141    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d3629c-1694-4cb8-86ad-46803f41dfc0-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.973795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.972137    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:05.996504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:05.993367    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:06.016547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.016478    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:06.016861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.016832    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:06.041178 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.041079    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7kt\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-kube-api-access-5p7kt\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:06.147328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.147238    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:06.429543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.429505    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:51:06.430050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.429566    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:51:06.524010 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.523977    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" event={"ID":"30f142a2-daa3-4115-876d-111e248aea92","Type":"ContainerStarted","Data":"32e5f97ba36284070ba3c43217d15d7f27b35f9da4c3ca87a1aa187733dc872a"}
Oct 29 22:51:06.629367 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.629338    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"]
Oct 29 22:51:06.631325 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:06.631284    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d3629c_1694_4cb8_86ad_46803f41dfc0.slice/crio-f3e9cb12af550c2f1dc9c8d0bd648f67a415bf932021c624219f11c402bbed0e WatchSource:0}: Error finding container f3e9cb12af550c2f1dc9c8d0bd648f67a415bf932021c624219f11c402bbed0e: Status 404 returned error can't find the container with id f3e9cb12af550c2f1dc9c8d0bd648f67a415bf932021c624219f11c402bbed0e
Oct 29 22:51:06.638040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.638000    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:06.638251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.638226    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:06.638388 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.638370    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:06.753201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.753125    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:51:06.759936 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.759662    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:51:06.759936 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.759724    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:51:06.807314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.807183    4827 patch_prober.go:29] interesting pod/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:06.807314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.807244    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:06.895969 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.895822    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"]
Oct 29 22:51:06.900790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:06.897874    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-639067f9-8btnk"]
Oct 29 22:51:07.399279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.399241    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:51:07.399587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.399537    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:51:07.545080 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.545046    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" event={"ID":"05d3629c-1694-4cb8-86ad-46803f41dfc0","Type":"ContainerStarted","Data":"9f1c4b6a33310223e33da26d618a1278935d44c8c5a175ab0c72ac269c333b9e"}
Oct 29 22:51:07.545691 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.545671    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" event={"ID":"05d3629c-1694-4cb8-86ad-46803f41dfc0","Type":"ContainerStarted","Data":"f3e9cb12af550c2f1dc9c8d0bd648f67a415bf932021c624219f11c402bbed0e"}
Oct 29 22:51:07.617791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.598180    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podStartSLOduration=2.598160424 podStartE2EDuration="2.598160424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:07.59614448 +0000 UTC m=+503.219055522" watchObservedRunningTime="2026-10-29 22:51:07.598160424 +0000 UTC m=+503.221071445"
Oct 29 22:51:07.756843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.753000    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:51:07.756843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.753059    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:51:07.812235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.812194    4827 patch_prober.go:29] interesting pod/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:07.812498 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:07.812470    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:08.148892 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.148854    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:08.153445 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.153410    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:08.202791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.201867    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:51:08.401846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.401147    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:51:08.401846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.401207    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:51:08.430271 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.430210    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"]
Oct 29 22:51:08.445792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.443949    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.480822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.463750    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"]
Oct 29 22:51:08.480822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.477319    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-17394a-openshift-default-dockercfg-x6t7f\""
Oct 29 22:51:08.538096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.538059    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d6dc6c-124b-47e5-a04a-0adb3a87d113" path="/var/lib/kubelet/pods/77d6dc6c-124b-47e5-a04a-0adb3a87d113/volumes"
Oct 29 22:51:08.594789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.574075    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:08.602788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.598912    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.635011    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698180    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698320    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698353    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698404    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698454    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698558    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698584    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6022bfd-dfbd-45c3-8209-5697f8517b36-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.699837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.698611    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtd9d\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-kube-api-access-dtd9d\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.754309 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.754199    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:51:08.754586 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.754555    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:51:08.822984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.822925    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.823235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.823215    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6022bfd-dfbd-45c3-8209-5697f8517b36-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.823366 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.823350    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtd9d\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-kube-api-access-dtd9d\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.823486 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.823471    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.823613 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.823595    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.823849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.823830    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.823980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.823965    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.824115 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.824098    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.824237 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.824223    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.825063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.825040    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.825912 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.825889    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6022bfd-dfbd-45c3-8209-5697f8517b36-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.826707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.826144    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.826852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.826475    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.827209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.827187    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.832848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.832185    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.837385 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.837357    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.849631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.849429    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtd9d\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-kube-api-access-dtd9d\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:08.850202 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:08.850161    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:09.073843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.073492    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:09.186220 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.186180    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:51:09.197791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.197364    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:51:09.406812 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.403198    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:51:09.406812 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.403262    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:51:09.516085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.515997    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"]
Oct 29 22:51:09.519539 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:09.519512    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6022bfd_dfbd_45c3_8209_5697f8517b36.slice/crio-bec8cdd2dfaeed11d462b70b8a9c2642813f85fc7d54bcc86d59c191098b068c WatchSource:0}: Error finding container bec8cdd2dfaeed11d462b70b8a9c2642813f85fc7d54bcc86d59c191098b068c: Status 404 returned error can't find the container with id bec8cdd2dfaeed11d462b70b8a9c2642813f85fc7d54bcc86d59c191098b068c
Oct 29 22:51:09.522323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.522290    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:09.522415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.522354    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:09.522415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.522383    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:09.661793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.660592    4827 generic.go:356] "Generic (PLEG): container finished" podID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerID="55550a5f6051bf7709660fda246d561a3a5ba264dc2a603bc25cfc83fa100bd1" exitCode=0
Oct 29 22:51:09.661793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.660684    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" event={"ID":"29113d99-6eaf-4046-b4d1-04318b7bdf38","Type":"ContainerDied","Data":"55550a5f6051bf7709660fda246d561a3a5ba264dc2a603bc25cfc83fa100bd1"}
Oct 29 22:51:09.661793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.660746    4827 scope.go:122] "RemoveContainer" containerID="da2d3341c3e6dddf6afc26dae0bae82e3c7cee8afcd3dac3ccb93727a8832adb"
Oct 29 22:51:09.718846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.703245    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" event={"ID":"e6022bfd-dfbd-45c3-8209-5697f8517b36","Type":"ContainerStarted","Data":"bec8cdd2dfaeed11d462b70b8a9c2642813f85fc7d54bcc86d59c191098b068c"}
Oct 29 22:51:09.743536 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.743505    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:51:09.753757 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.753721    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:51:09.753951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.753789    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759201    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-data\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759232    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-podinfo\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759285    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-token\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-token\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759347    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-envoy\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759388    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-socket\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759406    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-certs\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759419    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-data" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759438    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-kube-api-access-dbcqq\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-kube-api-access-dbcqq\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759479    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-credential-socket\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759501    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29113d99-6eaf-4046-b4d1-04318b7bdf38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29113d99-6eaf-4046-b4d1-04318b7bdf38-istiod-ca-cert\") pod \"29113d99-6eaf-4046-b4d1-04318b7bdf38\" (UID: \"29113d99-6eaf-4046-b4d1-04318b7bdf38\") "
Oct 29 22:51:09.759894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.759752    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.760882 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.760511    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-credential-socket" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:09.760882 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.760642    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-socket" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:09.760882 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.760846    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29113d99-6eaf-4046-b4d1-04318b7bdf38-istiod-ca-cert" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:51:09.761094 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.760901    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-certs" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:09.761932 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.761908    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-envoy" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:09.762678 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.762588    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-podinfo" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:51:09.772788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.772005    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-kube-api-access-dbcqq" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "kube-api-access-dbcqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:09.838810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.833969    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-token" pod "29113d99-6eaf-4046-b4d1-04318b7bdf38" (UID: "29113d99-6eaf-4046-b4d1-04318b7bdf38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860649    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860684    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860700    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860714    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbcqq\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-kube-api-access-dbcqq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860729    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29113d99-6eaf-4046-b4d1-04318b7bdf38-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860742    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29113d99-6eaf-4046-b4d1-04318b7bdf38-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860755    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.860803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:09.860779    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29113d99-6eaf-4046-b4d1-04318b7bdf38-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:10.218546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.218511    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:51:10.508406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.508382    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:51:10.508722 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.508703    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:51:10.509422 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.509404    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 22:51:10.710781 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.710726    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" event={"ID":"e6022bfd-dfbd-45c3-8209-5697f8517b36","Type":"ContainerStarted","Data":"84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a"}
Oct 29 22:51:10.712539 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.712510    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854" event={"ID":"29113d99-6eaf-4046-b4d1-04318b7bdf38","Type":"ContainerDied","Data":"2a57d1552fcf60b7535a5f8e72b4632b35762627c8263805c829f9a651e84227"}
Oct 29 22:51:10.712685 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.712547    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"
Oct 29 22:51:10.712685 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.712567    4827 scope.go:122] "RemoveContainer" containerID="55550a5f6051bf7709660fda246d561a3a5ba264dc2a603bc25cfc83fa100bd1"
Oct 29 22:51:10.753625 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.753596    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:51:10.753802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.753660    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:51:10.761489 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.761387    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podStartSLOduration=2.76137064 podStartE2EDuration="2.76137064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:10.760428632 +0000 UTC m=+506.383339652" watchObservedRunningTime="2026-10-29 22:51:10.76137064 +0000 UTC m=+506.384281662"
Oct 29 22:51:10.806872 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.806839    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"]
Oct 29 22:51:10.814689 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:10.814666    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-72818393-openshift-default-5f4b788dc5-86854"]
Oct 29 22:51:11.075447 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:11.075357    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:11.098137 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:11.098110    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:11.718804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:11.718779    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:11.719759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:11.719737    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:11.752600 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:11.752571    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:51:11.752724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:11.752615    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:51:12.500387 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:12.499906    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" path="/var/lib/kubelet/pods/29113d99-6eaf-4046-b4d1-04318b7bdf38/volumes"
Oct 29 22:51:12.753425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:12.753357    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:51:12.753794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:12.753444    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:51:13.756884 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:13.756855    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:51:13.757891 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:13.757873    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 22:51:15.251942 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.251907    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"]
Oct 29 22:51:15.252539 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.252520    4827 cpu_manager.go:414] "containerMap: removing container" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy"
Oct 29 22:51:15.252634 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.252541    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy"
Oct 29 22:51:15.252634 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.252569    4827 cpu_manager.go:414] "containerMap: removing container" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy"
Oct 29 22:51:15.252634 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.252577    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy"
Oct 29 22:51:15.252826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.252672    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy"
Oct 29 22:51:15.252826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.252686    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="29113d99-6eaf-4046-b4d1-04318b7bdf38" containerName="istio-proxy"
Oct 29 22:51:15.274403 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.270987    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.296870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.287205    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"]
Oct 29 22:51:15.309511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.301693    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-17394a\""
Oct 29 22:51:15.309511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.301950    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-17394a-dockercfg-8625j\""
Oct 29 22:51:15.404241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.404208    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfdc\" (UniqueName: \"kubernetes.io/projected/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-kube-api-access-xmfdc\") pod \"payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.404417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.404258    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.505811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.505710    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfdc\" (UniqueName: \"kubernetes.io/projected/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-kube-api-access-xmfdc\") pod \"payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.505811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.505754    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.506477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.506455    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.529277 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.529250    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfdc\" (UniqueName: \"kubernetes.io/projected/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-kube-api-access-xmfdc\") pod \"payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.649897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.649866    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:15.943841 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:15.943816    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"]
Oct 29 22:51:15.946179 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:15.946152    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dde070a_d0fa_4dd9_9fd4_eb1affab6cb1.slice/crio-933ab46da54106364bb04fc32d562cb94343b85bf9a5fb4641ad008da88f7f5b WatchSource:0}: Error finding container 933ab46da54106364bb04fc32d562cb94343b85bf9a5fb4641ad008da88f7f5b: Status 404 returned error can't find the container with id 933ab46da54106364bb04fc32d562cb94343b85bf9a5fb4641ad008da88f7f5b
Oct 29 22:51:16.743835 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:16.743786    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" event={"ID":"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1","Type":"ContainerStarted","Data":"e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8"}
Oct 29 22:51:16.744281 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:16.743854    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" event={"ID":"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1","Type":"ContainerStarted","Data":"933ab46da54106364bb04fc32d562cb94343b85bf9a5fb4641ad008da88f7f5b"}
Oct 29 22:51:16.744281 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:16.743899    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:16.789525 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:16.789465    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" podStartSLOduration=1.789447018 podStartE2EDuration="1.789447018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:16.788275864 +0000 UTC m=+512.411186885" watchObservedRunningTime="2026-10-29 22:51:16.789447018 +0000 UTC m=+512.412358039"
Oct 29 22:51:17.302356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:17.302314    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"]
Oct 29 22:51:17.306557 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:17.306530    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1db2136c-mm9gb"]
Oct 29 22:51:17.443298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:17.443266    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"]
Oct 29 22:51:17.443577 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:17.443529    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy" containerID="cri-o://06be3f0b35829355a9d8509fcc4dad9a125c9e245fc4145430705514f732cced" gracePeriod=30
Oct 29 22:51:18.487131 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:18.487098    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ff09f2-e001-43ed-9b90-8b5404432df4" path="/var/lib/kubelet/pods/05ff09f2-e001-43ed-9b90-8b5404432df4/volumes"
Oct 29 22:51:21.006337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:21.006302    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"]
Oct 29 22:51:21.017540 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:21.017510    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c314108-jjpwt"]
Oct 29 22:51:21.154683 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:21.154646    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"]
Oct 29 22:51:21.155025 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:21.154994    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" containerID="cri-o://5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd" gracePeriod=30
Oct 29 22:51:22.489224 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.489187    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca" path="/var/lib/kubelet/pods/d6b4a8a3-222e-48d1-bb6a-36ff3aed6bca/volumes"
Oct 29 22:51:22.536370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.536338    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"]
Oct 29 22:51:22.539211 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.539192    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.557511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.557482    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b2921b-openshift-default-dockercfg-652vx\""
Oct 29 22:51:22.578544 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.578483    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"]
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626392    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626508    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626570    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626613    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626689    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626721    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626782    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blk7\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-kube-api-access-8blk7\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626816    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.627124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.626924    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7201b82-d529-4613-bff3-68b8d8fbf266-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733485    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733537    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733576    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733619    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733693    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733727    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733790    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blk7\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-kube-api-access-8blk7\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733827    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.733938    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7201b82-d529-4613-bff3-68b8d8fbf266-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.734988    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7201b82-d529-4613-bff3-68b8d8fbf266-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.735833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.735268    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.754785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.749329    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.754785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.749597    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.769707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.760594    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.784323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.784254    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.836458 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.836418    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.838915 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.838862    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.840836 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.840488    4827 generic.go:356] "Generic (PLEG): container finished" podID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerID="06be3f0b35829355a9d8509fcc4dad9a125c9e245fc4145430705514f732cced" exitCode=0
Oct 29 22:51:22.840836 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.840625    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" event={"ID":"9ccae810-2b20-4331-993b-8e6c173bf3cc","Type":"ContainerDied","Data":"06be3f0b35829355a9d8509fcc4dad9a125c9e245fc4145430705514f732cced"}
Oct 29 22:51:22.840836 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.840682    4827 scope.go:122] "RemoveContainer" containerID="a9ca609f4b79b799c35f0b30e8a236d2e0022dde0a2c6ce40fdad6beedc416f3"
Oct 29 22:51:22.842202 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.842177    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blk7\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-kube-api-access-8blk7\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.853281 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.853249    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:22.908503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.908472    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.935973    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-data\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936065    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-kube-api-access-p4j8b\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-kube-api-access-p4j8b\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936113    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-credential-socket\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936155    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccae810-2b20-4331-993b-8e6c173bf3cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccae810-2b20-4331-993b-8e6c173bf3cc-istiod-ca-cert\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936173    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-token\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-token\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936191    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-certs\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936215    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-podinfo\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936234    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-envoy\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.936380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936286    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-socket\") pod \"9ccae810-2b20-4331-993b-8e6c173bf3cc\" (UID: \"9ccae810-2b20-4331-993b-8e6c173bf3cc\") "
Oct 29 22:51:22.937497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.936867    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-certs" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:22.937497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.937089    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-socket" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:22.937497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.937237    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccae810-2b20-4331-993b-8e6c173bf3cc-istiod-ca-cert" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:51:22.937497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.937485    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-credential-socket" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:22.937725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.937617    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-data" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:22.939999 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.939962    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-podinfo" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:51:22.941312 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.941279    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-kube-api-access-p4j8b" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "kube-api-access-p4j8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:22.941456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.941424    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-token" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:22.941538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:22.941495    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-envoy" pod "9ccae810-2b20-4331-993b-8e6c173bf3cc" (UID: "9ccae810-2b20-4331-993b-8e6c173bf3cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038006    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccae810-2b20-4331-993b-8e6c173bf3cc-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038037    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038052    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038064    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038075    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038088    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038100    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038113    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4j8b\" (UniqueName: \"kubernetes.io/projected/9ccae810-2b20-4331-993b-8e6c173bf3cc-kube-api-access-p4j8b\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.038164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.038126    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccae810-2b20-4331-993b-8e6c173bf3cc-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:23.308302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.307669    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"]
Oct 29 22:51:23.311130 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:23.311074    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7201b82_d529_4613_bff3_68b8d8fbf266.slice/crio-39c48962832ba4506f1d59e4af6497c9e12f0828ba747b5e92bda4198ba15f34 WatchSource:0}: Error finding container 39c48962832ba4506f1d59e4af6497c9e12f0828ba747b5e92bda4198ba15f34: Status 404 returned error can't find the container with id 39c48962832ba4506f1d59e4af6497c9e12f0828ba747b5e92bda4198ba15f34
Oct 29 22:51:23.316085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.315157    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:23.316085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.315302    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:23.316085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.315385    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:23.846889 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.846861    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"
Oct 29 22:51:23.847349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.846907    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx" event={"ID":"9ccae810-2b20-4331-993b-8e6c173bf3cc","Type":"ContainerDied","Data":"33d5dde1b4f0955a3cab0fe09bccf0091af979e1b8b83ca6d55d2273cef8bd75"}
Oct 29 22:51:23.847349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.846971    4827 scope.go:122] "RemoveContainer" containerID="06be3f0b35829355a9d8509fcc4dad9a125c9e245fc4145430705514f732cced"
Oct 29 22:51:23.848714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.848689    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" event={"ID":"a7201b82-d529-4613-bff3-68b8d8fbf266","Type":"ContainerStarted","Data":"f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b"}
Oct 29 22:51:23.848830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.848739    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" event={"ID":"a7201b82-d529-4613-bff3-68b8d8fbf266","Type":"ContainerStarted","Data":"39c48962832ba4506f1d59e4af6497c9e12f0828ba747b5e92bda4198ba15f34"}
Oct 29 22:51:23.854324 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.854305    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:23.911563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.911532    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"]
Oct 29 22:51:23.923337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.923312    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1db2136c-openshift-default-6d94bdc5b5-6zpdx"]
Oct 29 22:51:23.956868 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:23.956822    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podStartSLOduration=1.956807328 podStartE2EDuration="1.956807328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:23.956182993 +0000 UTC m=+519.579094014" watchObservedRunningTime="2026-10-29 22:51:23.956807328 +0000 UTC m=+519.579718345"
Oct 29 22:51:24.190166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.190128    4827 patch_prober.go:29] interesting pod/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:51:24.190339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.190180    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:51:24.496428 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.496353    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" path="/var/lib/kubelet/pods/9ccae810-2b20-4331-993b-8e6c173bf3cc/volumes"
Oct 29 22:51:24.845529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.845449    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"]
Oct 29 22:51:24.845912 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.845898    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy"
Oct 29 22:51:24.845971 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.845913    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy"
Oct 29 22:51:24.845971 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.845922    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy"
Oct 29 22:51:24.845971 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.845927    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy"
Oct 29 22:51:24.846064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.846002    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy"
Oct 29 22:51:24.846064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.846009    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccae810-2b20-4331-993b-8e6c173bf3cc" containerName="istio-proxy"
Oct 29 22:51:24.848973 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.848955    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:24.861468 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.861438    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:24.894195 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.894157    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-575d59\""
Oct 29 22:51:24.894377 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.894235    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-575d59-dockercfg-vqbj5\""
Oct 29 22:51:24.895669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.895639    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"]
Oct 29 22:51:24.899667 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.898039    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feefe299-2d10-4139-a3b1-29a6dc9a68c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:24.899667 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.898117    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthgh\" (UniqueName: \"kubernetes.io/projected/feefe299-2d10-4139-a3b1-29a6dc9a68c2-kube-api-access-mthgh\") pod \"payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:24.900019 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.899998    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:24.901095 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:24.901078    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:25.044902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.007879    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthgh\" (UniqueName: \"kubernetes.io/projected/feefe299-2d10-4139-a3b1-29a6dc9a68c2-kube-api-access-mthgh\") pod \"payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:25.044902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.008065    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feefe299-2d10-4139-a3b1-29a6dc9a68c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:25.044902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.044460    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feefe299-2d10-4139-a3b1-29a6dc9a68c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:25.061127 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.061096    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthgh\" (UniqueName: \"kubernetes.io/projected/feefe299-2d10-4139-a3b1-29a6dc9a68c2-kube-api-access-mthgh\") pod \"payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:25.188068 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.187989    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:25.497212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.497184    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"]
Oct 29 22:51:25.498811 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:25.498756    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeefe299_2d10_4139_a3b1_29a6dc9a68c2.slice/crio-3d2aa064c105887129e1a6e3f7de103637edf154f390064e153ab069b4aab884 WatchSource:0}: Error finding container 3d2aa064c105887129e1a6e3f7de103637edf154f390064e153ab069b4aab884: Status 404 returned error can't find the container with id 3d2aa064c105887129e1a6e3f7de103637edf154f390064e153ab069b4aab884
Oct 29 22:51:25.905993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.905957    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" event={"ID":"feefe299-2d10-4139-a3b1-29a6dc9a68c2","Type":"ContainerStarted","Data":"cc8189589deaba35c57b7cd4a97b7927c789cacec26685fae8ad10c8488a30e4"}
Oct 29 22:51:25.906465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.906017    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" event={"ID":"feefe299-2d10-4139-a3b1-29a6dc9a68c2","Type":"ContainerStarted","Data":"3d2aa064c105887129e1a6e3f7de103637edf154f390064e153ab069b4aab884"}
Oct 29 22:51:25.947297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:25.947239    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" podStartSLOduration=1.947222468 podStartE2EDuration="1.947222468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:25.946344363 +0000 UTC m=+521.569255386" watchObservedRunningTime="2026-10-29 22:51:25.947222468 +0000 UTC m=+521.570133491"
Oct 29 22:51:26.492159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.492134    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528087    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-credential-socket\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528167    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-token\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-token\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528225    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-socket\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528275    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-envoy\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528339    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-podinfo\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528385    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-certs\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528435    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-data\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528482    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-kube-api-access-bz76k\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-kube-api-access-bz76k\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.528538    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istiod-ca-cert\") pod \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\" (UID: \"c3b78bb5-b3ed-4514-96ba-167bdb074da5\") "
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.533706    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-certs" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.533790    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-socket" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.533826    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-data" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:26.549908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.534090    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-credential-socket" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:26.550677 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.550340    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istiod-ca-cert" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:51:26.557888 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.555951    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-token" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:26.557888 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.555970    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-kube-api-access-bz76k" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "kube-api-access-bz76k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:26.558092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.557957    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-envoy" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:26.562213 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.562169    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-podinfo" pod "c3b78bb5-b3ed-4514-96ba-167bdb074da5" (UID: "c3b78bb5-b3ed-4514-96ba-167bdb074da5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:51:26.630464 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630430    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630464 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630463    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630464 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630472    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630481    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630489    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630498    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz76k\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-kube-api-access-bz76k\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630507    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630515    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b78bb5-b3ed-4514-96ba-167bdb074da5-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.630823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.630523    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b78bb5-b3ed-4514-96ba-167bdb074da5-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:26.912728 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.912692    4827 generic.go:356] "Generic (PLEG): container finished" podID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerID="5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd" exitCode=0
Oct 29 22:51:26.913284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.912744    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"
Oct 29 22:51:26.913284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.912804    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" event={"ID":"c3b78bb5-b3ed-4514-96ba-167bdb074da5","Type":"ContainerDied","Data":"5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd"}
Oct 29 22:51:26.913284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.912858    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k" event={"ID":"c3b78bb5-b3ed-4514-96ba-167bdb074da5","Type":"ContainerDied","Data":"cccfba126eee0f455595855ec44e74c98cd0125332a7144630e1597a671f6b20"}
Oct 29 22:51:26.913284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.912888    4827 scope.go:122] "RemoveContainer" containerID="5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd"
Oct 29 22:51:26.913469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.913343    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:26.922462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.922446    4827 scope.go:122] "RemoveContainer" containerID="a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a"
Oct 29 22:51:26.932120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.932104    4827 scope.go:122] "RemoveContainer" containerID="5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd"
Oct 29 22:51:26.932397 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:51:26.932378    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd\": container with ID starting with 5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd not found: ID does not exist" containerID="5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd"
Oct 29 22:51:26.932456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.932417    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd"} err="failed to get container status \"5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd\": rpc error: code = NotFound desc = could not find container \"5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd\": container with ID starting with 5c0767f1cb052dcf9bb4677bd49e95ea78cdde706bc4748c2f2dcbf7b5aa00fd not found: ID does not exist"
Oct 29 22:51:26.932456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.932440    4827 scope.go:122] "RemoveContainer" containerID="a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a"
Oct 29 22:51:26.932688 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:51:26.932672    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a\": container with ID starting with a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a not found: ID does not exist" containerID="a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a"
Oct 29 22:51:26.932746 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.932699    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a"} err="failed to get container status \"a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a\": rpc error: code = NotFound desc = could not find container \"a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a\": container with ID starting with a915e39883bc2bd90ce5c69ea9baee591e76a21f8da214735c61b85511edbe5a not found: ID does not exist"
Oct 29 22:51:26.987919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.987839    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"]
Oct 29 22:51:26.989803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:26.989681    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3c314108-openshift-default-7dc49f84b5-hnn8k"]
Oct 29 22:51:27.752091 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:27.752061    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 22:51:28.281562 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.281528    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"]
Oct 29 22:51:28.282381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.282358    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy"
Oct 29 22:51:28.282381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.282381    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy"
Oct 29 22:51:28.282559 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.282403    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy"
Oct 29 22:51:28.282559 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.282412    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy"
Oct 29 22:51:28.282559 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.282509    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy"
Oct 29 22:51:28.282559 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.282524    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" containerName="istio-proxy"
Oct 29 22:51:28.287732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.287711    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.302926 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.302907    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-b2921b\""
Oct 29 22:51:28.314900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.314874    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-b2921b-dockercfg-xd48j\""
Oct 29 22:51:28.317982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.317962    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"]
Oct 29 22:51:28.347897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.347688    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xn8\" (UniqueName: \"kubernetes.io/projected/a9db7699-245e-4237-9301-096107be2fa2-kube-api-access-m4xn8\") pod \"payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.347897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.347809    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db7699-245e-4237-9301-096107be2fa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.389792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.369725    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"]
Oct 29 22:51:28.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.399093    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"]
Oct 29 22:51:28.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.399214    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.449749 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.449718    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7zv\" (UniqueName: \"kubernetes.io/projected/2c0abba1-df13-4539-98f7-1c32afacef23-kube-api-access-kk7zv\") pod \"payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.449944 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.449879    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xn8\" (UniqueName: \"kubernetes.io/projected/a9db7699-245e-4237-9301-096107be2fa2-kube-api-access-m4xn8\") pod \"payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.449944 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.449928    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0abba1-df13-4539-98f7-1c32afacef23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.450067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.449972    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db7699-245e-4237-9301-096107be2fa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.450742 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.450717    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db7699-245e-4237-9301-096107be2fa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.476262 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.476233    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xn8\" (UniqueName: \"kubernetes.io/projected/a9db7699-245e-4237-9301-096107be2fa2-kube-api-access-m4xn8\") pod \"payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.487447 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.487417    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b78bb5-b3ed-4514-96ba-167bdb074da5" path="/var/lib/kubelet/pods/c3b78bb5-b3ed-4514-96ba-167bdb074da5/volumes"
Oct 29 22:51:28.550923 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.550846    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0abba1-df13-4539-98f7-1c32afacef23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.551070 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.550929    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7zv\" (UniqueName: \"kubernetes.io/projected/2c0abba1-df13-4539-98f7-1c32afacef23-kube-api-access-kk7zv\") pod \"payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.551505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.551485    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0abba1-df13-4539-98f7-1c32afacef23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.574987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.574958    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7zv\" (UniqueName: \"kubernetes.io/projected/2c0abba1-df13-4539-98f7-1c32afacef23-kube-api-access-kk7zv\") pod \"payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.617287 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.617245    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:28.755183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.755149    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:28.922711 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:28.922680    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"]
Oct 29 22:51:28.924225 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:28.924196    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9db7699_245e_4237_9301_096107be2fa2.slice/crio-a42083102132b9052bd127eb86f97ca1d0485c70da2002d3ca7a1e52aa5652a4 WatchSource:0}: Error finding container a42083102132b9052bd127eb86f97ca1d0485c70da2002d3ca7a1e52aa5652a4: Status 404 returned error can't find the container with id a42083102132b9052bd127eb86f97ca1d0485c70da2002d3ca7a1e52aa5652a4
Oct 29 22:51:29.064508 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.064457    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"]
Oct 29 22:51:29.067442 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:29.067400    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0abba1_df13_4539_98f7_1c32afacef23.slice/crio-386c5bbaeda9c56894aeeb46a5953a062d482ca4d1386da883f9f46e57fb0a16 WatchSource:0}: Error finding container 386c5bbaeda9c56894aeeb46a5953a062d482ca4d1386da883f9f46e57fb0a16: Status 404 returned error can't find the container with id 386c5bbaeda9c56894aeeb46a5953a062d482ca4d1386da883f9f46e57fb0a16
Oct 29 22:51:29.935384 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.935348    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" event={"ID":"a9db7699-245e-4237-9301-096107be2fa2","Type":"ContainerStarted","Data":"d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c"}
Oct 29 22:51:29.935887 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.935500    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:29.935887 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.935524    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" event={"ID":"a9db7699-245e-4237-9301-096107be2fa2","Type":"ContainerStarted","Data":"a42083102132b9052bd127eb86f97ca1d0485c70da2002d3ca7a1e52aa5652a4"}
Oct 29 22:51:29.937132 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.937107    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc" event={"ID":"2c0abba1-df13-4539-98f7-1c32afacef23","Type":"ContainerStarted","Data":"6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2"}
Oct 29 22:51:29.937259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.937153    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc" event={"ID":"2c0abba1-df13-4539-98f7-1c32afacef23","Type":"ContainerStarted","Data":"386c5bbaeda9c56894aeeb46a5953a062d482ca4d1386da883f9f46e57fb0a16"}
Oct 29 22:51:29.937259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.937175    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:29.986191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:29.986139    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" podStartSLOduration=1.9861263930000002 podStartE2EDuration="1.986126393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:29.985530706 +0000 UTC m=+525.608441722" watchObservedRunningTime="2026-10-29 22:51:29.986126393 +0000 UTC m=+525.609037465"
Oct 29 22:51:30.032414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:30.027330    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc" podStartSLOduration=2.027311747 podStartE2EDuration="2.027311747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:30.025739244 +0000 UTC m=+525.648650293" watchObservedRunningTime="2026-10-29 22:51:30.027311747 +0000 UTC m=+525.650222769"
Oct 29 22:51:31.949676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:31.949648    4827 generic.go:356] "Generic (PLEG): container finished" podID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerID="9f1c4b6a33310223e33da26d618a1278935d44c8c5a175ab0c72ac269c333b9e" exitCode=0
Oct 29 22:51:31.950108 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:31.949716    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" event={"ID":"05d3629c-1694-4cb8-86ad-46803f41dfc0","Type":"ContainerDied","Data":"9f1c4b6a33310223e33da26d618a1278935d44c8c5a175ab0c72ac269c333b9e"}
Oct 29 22:51:31.950189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:31.950172    4827 scope.go:122] "RemoveContainer" containerID="9f1c4b6a33310223e33da26d618a1278935d44c8c5a175ab0c72ac269c333b9e"
Oct 29 22:51:31.951819 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:31.951760    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:31.951943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:31.951872    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:31.951943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:31.951917    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:32.834500 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:51:32.834467    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:51:32.957602 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:32.957568    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" event={"ID":"05d3629c-1694-4cb8-86ad-46803f41dfc0","Type":"ContainerStarted","Data":"5c900aef2c40123ee0eadddddd5868e597af802f399de37304babc94b7db3f78"}
Oct 29 22:51:33.148319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:33.148294    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:33.148531 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:33.148381    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 22:51:33.148531 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:33.148416    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 22:51:34.148190 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:34.148152    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 22:51:34.148909 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:34.148873    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 22:51:35.148650 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:35.148620    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 22:51:35.149067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:35.148669    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 22:51:36.148512 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:36.148478    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 22:51:36.148683 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:36.148532    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 22:51:37.148333 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:37.148295    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 22:51:37.148504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:37.148365    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 22:51:37.922825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:37.922734    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 22:51:38.274411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:38.274318    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:38.274697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:38.274671    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:38.275863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:38.275835    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 22:51:40.034089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.034003    4827 generic.go:356] "Generic (PLEG): container finished" podID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerID="84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a" exitCode=0
Oct 29 22:51:40.034652 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.034629    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" event={"ID":"e6022bfd-dfbd-45c3-8209-5697f8517b36","Type":"ContainerDied","Data":"84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a"}
Oct 29 22:51:40.035336 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.035315    4827 scope.go:122] "RemoveContainer" containerID="84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a"
Oct 29 22:51:40.037153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.037123    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:40.037264 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.037210    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:40.037328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.037261    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:40.945318 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.945287    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 22:51:40.945483 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:40.945378    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 22:51:41.041282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:41.041250    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" event={"ID":"e6022bfd-dfbd-45c3-8209-5697f8517b36","Type":"ContainerStarted","Data":"2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593"}
Oct 29 22:51:41.091156 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:41.091115    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:41.092162 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:41.092128    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 29 22:51:41.092296 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:41.092188    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 29 22:51:41.719567 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:41.719531    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:42.079251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:42.079160    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 29 22:51:42.079877 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:42.079846    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 29 22:51:43.074552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:43.074130    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 29 22:51:43.074552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:43.074201    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 29 22:51:44.078197 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:44.078161    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 29 22:51:44.078848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:44.078812    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 29 22:51:45.084797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:45.082007    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 29 22:51:45.084797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:45.082078    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 29 22:51:46.089366 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:46.089328    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 29 22:51:46.089909 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:46.089376    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 29 22:51:47.078254 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.078222    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:47.079389 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.079371    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 22:51:47.267884 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.267852    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"]
Oct 29 22:51:47.272067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.272043    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.309400 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.309379    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"kube-root-ca.crt\""
Oct 29 22:51:47.314060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.314036    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"premium-w1-e131af-kserve-self-signed-certs\""
Oct 29 22:51:47.314256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.314236    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"default-dockercfg-xwjmb\""
Oct 29 22:51:47.316864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.316840    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"]
Oct 29 22:51:47.341997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.341926    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"openshift-service-ca.crt\""
Oct 29 22:51:47.348377 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.348351    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-dshm\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.348505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.348400    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-tmp-dir\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.348546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.348532    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-model-cache\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.348583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.348557    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705240b9-b2bf-4a99-b78e-6216e4e0350e-tls-certs\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.348631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.348582    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-home\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.348631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.348602    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trd9\" (UniqueName: \"kubernetes.io/projected/705240b9-b2bf-4a99-b78e-6216e4e0350e-kube-api-access-8trd9\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449292 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449258    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-model-cache\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449292 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449297    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705240b9-b2bf-4a99-b78e-6216e4e0350e-tls-certs\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449556 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449327    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-home\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449556 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449358    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trd9\" (UniqueName: \"kubernetes.io/projected/705240b9-b2bf-4a99-b78e-6216e4e0350e-kube-api-access-8trd9\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449556 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449426    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-dshm\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449556 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449485    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-tmp-dir\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449735    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-model-cache\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449874 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449815    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-home\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.449874 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.449840    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-tmp-dir\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.451626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.451605    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-dshm\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.452058 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.452037    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705240b9-b2bf-4a99-b78e-6216e4e0350e-tls-certs\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.472396 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.472364    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trd9\" (UniqueName: \"kubernetes.io/projected/705240b9-b2bf-4a99-b78e-6216e4e0350e-kube-api-access-8trd9\") pod \"premium-w1-e131af-kserve-7dbccf6cd6-ndmm8\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") " pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.585252 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.585187    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:47.887387 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:47.887366    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"]
Oct 29 22:51:47.889571 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:47.889539    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod705240b9_b2bf_4a99_b78e_6216e4e0350e.slice/crio-70351c9e813e27d149bcf21cb3bf6a1925c40165425888c6ddbf67e4a68e7e78 WatchSource:0}: Error finding container 70351c9e813e27d149bcf21cb3bf6a1925c40165425888c6ddbf67e4a68e7e78: Status 404 returned error can't find the container with id 70351c9e813e27d149bcf21cb3bf6a1925c40165425888c6ddbf67e4a68e7e78
Oct 29 22:51:48.100013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.099979    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" event={"ID":"705240b9-b2bf-4a99-b78e-6216e4e0350e","Type":"ContainerStarted","Data":"f2b56a6b983165a972ac8baeb3ace6d9121d54d8eb54bb6f9ad207cbd2c95dfb"}
Oct 29 22:51:48.100178 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.100022    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" event={"ID":"705240b9-b2bf-4a99-b78e-6216e4e0350e","Type":"ContainerStarted","Data":"70351c9e813e27d149bcf21cb3bf6a1925c40165425888c6ddbf67e4a68e7e78"}
Oct 29 22:51:48.141483 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.141375    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podStartSLOduration=1.141359031 podStartE2EDuration="1.141359031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:48.140123267 +0000 UTC m=+543.763034285" watchObservedRunningTime="2026-10-29 22:51:48.141359031 +0000 UTC m=+543.764270051"
Oct 29 22:51:48.586977 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.586876    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:48.834198 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.832827    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"]
Oct 29 22:51:48.843834 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.841077    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:48.854970 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.854943    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"]
Oct 29 22:51:48.862096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.861670    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:51:48.862096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.861808    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:51:48.987424 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.987384    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jw5p\" (UniqueName: \"kubernetes.io/projected/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-kube-api-access-4jw5p\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:48.987789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.987745    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:48.987995 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:48.987967    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.089356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.089314    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.089356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.089357    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.089604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.089436    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jw5p\" (UniqueName: \"kubernetes.io/projected/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-kube-api-access-4jw5p\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.090122 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.090097    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.091619 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.091594    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.113001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.112937    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jw5p\" (UniqueName: \"kubernetes.io/projected/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-kube-api-access-4jw5p\") pod \"maas-api-revoke-sub-451bd927-pwqql\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.205281 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.205248    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:49.505111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:49.505088    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"]
Oct 29 22:51:49.507053 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:51:49.507029    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2ac21b_3f72_42d7_88e3_5cbe0f5f9627.slice/crio-6053d1ffeb73a5e3bfdb5eec5bf60f8fefb0440afa9bdea7841816dae6ec6246 WatchSource:0}: Error finding container 6053d1ffeb73a5e3bfdb5eec5bf60f8fefb0440afa9bdea7841816dae6ec6246: Status 404 returned error can't find the container with id 6053d1ffeb73a5e3bfdb5eec5bf60f8fefb0440afa9bdea7841816dae6ec6246
Oct 29 22:51:50.112571 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:50.112538    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql" event={"ID":"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627","Type":"ContainerStarted","Data":"1466394bd06ec48b079632ced2a94bbd85a151801aff81e43bbdff7f77dc00c9"}
Oct 29 22:51:50.113032 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:50.112600    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql" event={"ID":"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627","Type":"ContainerStarted","Data":"6053d1ffeb73a5e3bfdb5eec5bf60f8fefb0440afa9bdea7841816dae6ec6246"}
Oct 29 22:51:50.154159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:50.154066    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql" podStartSLOduration=2.15404738 podStartE2EDuration="2.15404738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:50.151908902 +0000 UTC m=+545.774819923" watchObservedRunningTime="2026-10-29 22:51:50.15404738 +0000 UTC m=+545.776958400"
Oct 29 22:51:51.119478 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.119444    4827 generic.go:356] "Generic (PLEG): container finished" podID="db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" containerID="1466394bd06ec48b079632ced2a94bbd85a151801aff81e43bbdff7f77dc00c9" exitCode=0
Oct 29 22:51:51.119920 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.119533    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql" event={"ID":"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627","Type":"ContainerDied","Data":"1466394bd06ec48b079632ced2a94bbd85a151801aff81e43bbdff7f77dc00c9"}
Oct 29 22:51:51.792711 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.792674    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"]
Oct 29 22:51:51.805483 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.805453    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.821342 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.821318    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"]
Oct 29 22:51:51.826634 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.826609    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-model-cache\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.826804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.826664    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7gs\" (UniqueName: \"kubernetes.io/projected/4a0f92f1-7de0-4ba3-a324-415985263db5-kube-api-access-kf7gs\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.826863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.826811    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-home\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.826863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.826848    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-dshm\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.826961 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.826905    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f92f1-7de0-4ba3-a324-415985263db5-tls-certs\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.826961 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.826948    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-tmp-dir\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.831218 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.831185    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"openshift-service-ca.crt\""
Oct 29 22:51:51.831353 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.831227    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"kube-root-ca.crt\""
Oct 29 22:51:51.832117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.832093    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"default-dockercfg-8h2pz\""
Oct 29 22:51:51.832117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.832116    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"simulator-w4-063b46-kserve-self-signed-certs\""
Oct 29 22:51:51.927964 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.927930    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f92f1-7de0-4ba3-a324-415985263db5-tls-certs\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.927989    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-tmp-dir\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.928034    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-model-cache\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.928087    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7gs\" (UniqueName: \"kubernetes.io/projected/4a0f92f1-7de0-4ba3-a324-415985263db5-kube-api-access-kf7gs\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.928142    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-home\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.928177    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-dshm\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928597 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.928482    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-tmp-dir\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928597 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.928543    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-model-cache\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.928597 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.928571    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-home\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.930653 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.930625    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-dshm\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.930759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.930747    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f92f1-7de0-4ba3-a324-415985263db5-tls-certs\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:51.951301 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:51.951276    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7gs\" (UniqueName: \"kubernetes.io/projected/4a0f92f1-7de0-4ba3-a324-415985263db5-kube-api-access-kf7gs\") pod \"simulator-w4-063b46-kserve-5b46c8fb64-q5vl8\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") " pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:52.118194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.118107    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:52.321300 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.321271    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:52.333533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.333500    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-openshift-ca\") pod \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") "
Oct 29 22:51:52.333700 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.333567    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-xks-ca\") pod \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") "
Oct 29 22:51:52.333700 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.333673    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-kube-api-access-4jw5p\" (UniqueName: \"kubernetes.io/projected/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-kube-api-access-4jw5p\") pod \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\" (UID: \"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627\") "
Oct 29 22:51:52.334277 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.334247    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-openshift-ca" pod "db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" (UID: "db2ac21b-3f72-42d7-88e3-5cbe0f5f9627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:51:52.335950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.335918    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-xks-ca" pod "db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" (UID: "db2ac21b-3f72-42d7-88e3-5cbe0f5f9627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:51:52.336249 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.336224    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-kube-api-access-4jw5p" pod "db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" (UID: "db2ac21b-3f72-42d7-88e3-5cbe0f5f9627"). InnerVolumeSpecName "kube-api-access-4jw5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:52.435111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.435080    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:52.435111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.435106    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:52.435111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.435115    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jw5p\" (UniqueName: \"kubernetes.io/projected/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627-kube-api-access-4jw5p\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:51:52.497079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:52.497057    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"]
Oct 29 22:51:53.140006 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:53.139971    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" event={"ID":"4a0f92f1-7de0-4ba3-a324-415985263db5","Type":"ContainerStarted","Data":"ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af"}
Oct 29 22:51:53.140267 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:53.140238    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" event={"ID":"4a0f92f1-7de0-4ba3-a324-415985263db5","Type":"ContainerStarted","Data":"09e47fad63844474767ca91b2c87557b98141baacb432bb218309c909749a659"}
Oct 29 22:51:53.141696 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:53.141676    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql" event={"ID":"db2ac21b-3f72-42d7-88e3-5cbe0f5f9627","Type":"ContainerDied","Data":"6053d1ffeb73a5e3bfdb5eec5bf60f8fefb0440afa9bdea7841816dae6ec6246"}
Oct 29 22:51:53.141822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:53.141715    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"
Oct 29 22:51:53.141822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:53.141717    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6053d1ffeb73a5e3bfdb5eec5bf60f8fefb0440afa9bdea7841816dae6ec6246"
Oct 29 22:51:53.185788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:53.185456    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podStartSLOduration=2.185437366 podStartE2EDuration="2.185437366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:53.183150092 +0000 UTC m=+548.806061112" watchObservedRunningTime="2026-10-29 22:51:53.185437366 +0000 UTC m=+548.808348387"
Oct 29 22:51:54.120128 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:54.120095    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:51:55.941900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:55.941869    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:51:55.942365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:55.941922    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:51:57.585594 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:57.585560    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:57.598074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:57.598016    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:58.170971 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:58.170938    4827 generic.go:356] "Generic (PLEG): container finished" podID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerID="f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b" exitCode=0
Oct 29 22:51:58.171225 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:58.171013    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" event={"ID":"a7201b82-d529-4613-bff3-68b8d8fbf266","Type":"ContainerDied","Data":"f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b"}
Oct 29 22:51:58.171438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:58.171417    4827 scope.go:122] "RemoveContainer" containerID="f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b"
Oct 29 22:51:58.172758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:58.172729    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:58.172897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:58.172876    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:58.172971 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:58.172932    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:51:58.182967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:58.182947    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:51:59.201265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:59.201214    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" event={"ID":"a7201b82-d529-4613-bff3-68b8d8fbf266","Type":"ContainerStarted","Data":"19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308"}
Oct 29 22:51:59.854080 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:59.854046    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:51:59.854255 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:59.854227    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 29 22:51:59.854304 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:51:59.854278    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 29 22:52:00.854094 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:00.854061    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 29 22:52:00.854475 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:00.854120    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 29 22:52:01.854475 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:01.854431    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 29 22:52:01.854940 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:01.854499    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 29 22:52:02.120716 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:02.120638    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:52:02.133679 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:02.133658    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:52:02.227652 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:02.227620    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:52:02.823363 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:52:02.823327    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:52:02.853859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:02.853826    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 29 22:52:02.854044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:02.853879    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 29 22:52:03.291165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.291123    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"]
Oct 29 22:52:03.291626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.291610    4827 cpu_manager.go:414] "containerMap: removing container" podUID="db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" containerName="revoke-keys"
Oct 29 22:52:03.291626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.291626    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" containerName="revoke-keys"
Oct 29 22:52:03.291733 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.291723    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" containerName="revoke-keys"
Oct 29 22:52:03.295283 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.295260    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.320127 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.320105    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:52:03.329353 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.329324    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:52:03.331967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.331882    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"]
Oct 29 22:52:03.413087 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.413046    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8qr\" (UniqueName: \"kubernetes.io/projected/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-kube-api-access-kk8qr\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.413257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.413113    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.413338 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.413316    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.514780 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.514733    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.514929 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.514822    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8qr\" (UniqueName: \"kubernetes.io/projected/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-kube-api-access-kk8qr\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.514929 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.514870    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.515509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.515486    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.517257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.517238    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.537962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.537941    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8qr\" (UniqueName: \"kubernetes.io/projected/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-kube-api-access-kk8qr\") pod \"maas-api-revoke-sub-3ad608fc-mt4g9\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.608495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.608425    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:03.854337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.854304    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 29 22:52:03.854508 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.854355    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 29 22:52:03.907667 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:03.907632    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"]
Oct 29 22:52:03.909405 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:03.909365    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99b86c5_f967_4a0a_8ea2_c3d839a7f914.slice/crio-f38214cc445b6dce1ea09fd65bda3044aae55919c8bd6a29c5dee1fcdb7772c2 WatchSource:0}: Error finding container f38214cc445b6dce1ea09fd65bda3044aae55919c8bd6a29c5dee1fcdb7772c2: Status 404 returned error can't find the container with id f38214cc445b6dce1ea09fd65bda3044aae55919c8bd6a29c5dee1fcdb7772c2
Oct 29 22:52:04.234201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:04.234168    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9" event={"ID":"e99b86c5-f967-4a0a-8ea2-c3d839a7f914","Type":"ContainerStarted","Data":"f942fa2ea6b204d1c41c14a99a3f1d02d1f94ebe2996a20a23517b0c20782eea"}
Oct 29 22:52:04.234398 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:04.234226    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9" event={"ID":"e99b86c5-f967-4a0a-8ea2-c3d839a7f914","Type":"ContainerStarted","Data":"f38214cc445b6dce1ea09fd65bda3044aae55919c8bd6a29c5dee1fcdb7772c2"}
Oct 29 22:52:04.276209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:04.276155    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9" podStartSLOduration=1.276140667 podStartE2EDuration="1.276140667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:04.273743958 +0000 UTC m=+559.896654977" watchObservedRunningTime="2026-10-29 22:52:04.276140667 +0000 UTC m=+559.899051686"
Oct 29 22:52:04.858155 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:04.858119    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:52:04.858557 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:04.858416    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:52:04.859209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:04.859190    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 22:52:05.240562 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:05.240523    4827 generic.go:356] "Generic (PLEG): container finished" podID="e99b86c5-f967-4a0a-8ea2-c3d839a7f914" containerID="f942fa2ea6b204d1c41c14a99a3f1d02d1f94ebe2996a20a23517b0c20782eea" exitCode=0
Oct 29 22:52:05.240759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:05.240612    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9" event={"ID":"e99b86c5-f967-4a0a-8ea2-c3d839a7f914","Type":"ContainerDied","Data":"f942fa2ea6b204d1c41c14a99a3f1d02d1f94ebe2996a20a23517b0c20782eea"}
Oct 29 22:52:06.386560 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.386536    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:06.480000 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.479967    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"]
Oct 29 22:52:06.480586 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.480563    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e99b86c5-f967-4a0a-8ea2-c3d839a7f914" containerName="revoke-keys"
Oct 29 22:52:06.480688 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.480588    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99b86c5-f967-4a0a-8ea2-c3d839a7f914" containerName="revoke-keys"
Oct 29 22:52:06.480819 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.480737    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99b86c5-f967-4a0a-8ea2-c3d839a7f914" containerName="revoke-keys"
Oct 29 22:52:06.484438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.484409    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.502166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.502096    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"kube-root-ca.crt\""
Oct 29 22:52:06.502290 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.502190    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"openshift-service-ca.crt\""
Oct 29 22:52:06.502290 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.502267    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"simulator-w2-575d59-kserve-self-signed-certs\""
Oct 29 22:52:06.502809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.502196    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"default-dockercfg-stszs\""
Oct 29 22:52:06.505817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.505790    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"]
Oct 29 22:52:06.552681 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.552652    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-xks-ca\") pod \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") "
Oct 29 22:52:06.552869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.552835    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-kube-api-access-kk8qr\" (UniqueName: \"kubernetes.io/projected/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-kube-api-access-kk8qr\") pod \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") "
Oct 29 22:52:06.552935 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.552917    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-openshift-ca\") pod \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\" (UID: \"e99b86c5-f967-4a0a-8ea2-c3d839a7f914\") "
Oct 29 22:52:06.553262 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.553241    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-openshift-ca" pod "e99b86c5-f967-4a0a-8ea2-c3d839a7f914" (UID: "e99b86c5-f967-4a0a-8ea2-c3d839a7f914"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:06.554710 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.554683    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-xks-ca" pod "e99b86c5-f967-4a0a-8ea2-c3d839a7f914" (UID: "e99b86c5-f967-4a0a-8ea2-c3d839a7f914"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:06.554984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.554966    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-kube-api-access-kk8qr" pod "e99b86c5-f967-4a0a-8ea2-c3d839a7f914" (UID: "e99b86c5-f967-4a0a-8ea2-c3d839a7f914"). InnerVolumeSpecName "kube-api-access-kk8qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:06.653828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.653757    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tmp-dir\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.653979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.653858    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7j9\" (UniqueName: \"kubernetes.io/projected/8f5e2cbd-6e8f-47e0-a282-743b6525c169-kube-api-access-xn7j9\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.653979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.653909    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-dshm\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.654050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.653997    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-model-cache\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.654050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.654015    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-home\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.654050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.654032    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tls-certs\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.654143 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.654123    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:06.654143 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.654135    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8qr\" (UniqueName: \"kubernetes.io/projected/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-kube-api-access-kk8qr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:06.654208 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.654143    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99b86c5-f967-4a0a-8ea2-c3d839a7f914-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:06.755408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755325    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-dshm\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.755408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755396    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-model-cache\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.755648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755424    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-home\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.755648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755448    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tls-certs\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.755648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755541    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tmp-dir\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.755648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755596    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7j9\" (UniqueName: \"kubernetes.io/projected/8f5e2cbd-6e8f-47e0-a282-743b6525c169-kube-api-access-xn7j9\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.755958 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755932    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-model-cache\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.756010 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.755932    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-home\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.756173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.756149    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tmp-dir\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.758047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.758017    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-dshm\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.758166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.758148    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tls-certs\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.778955 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.778929    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7j9\" (UniqueName: \"kubernetes.io/projected/8f5e2cbd-6e8f-47e0-a282-743b6525c169-kube-api-access-xn7j9\") pod \"simulator-w2-575d59-kserve-856dbcbdfb-kdc2p\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") " pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:06.798813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:06.798792    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:07.099532 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.099505    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"]
Oct 29 22:52:07.101931 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:07.101888    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f5e2cbd_6e8f_47e0_a282_743b6525c169.slice/crio-8de9d4b45c117c89bd01ce6c33ab0971460cb1e0094c91d586ea7a186ea8e005 WatchSource:0}: Error finding container 8de9d4b45c117c89bd01ce6c33ab0971460cb1e0094c91d586ea7a186ea8e005: Status 404 returned error can't find the container with id 8de9d4b45c117c89bd01ce6c33ab0971460cb1e0094c91d586ea7a186ea8e005
Oct 29 22:52:07.256784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.256745    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"
Oct 29 22:52:07.257864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.257595    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9" event={"ID":"e99b86c5-f967-4a0a-8ea2-c3d839a7f914","Type":"ContainerDied","Data":"f38214cc445b6dce1ea09fd65bda3044aae55919c8bd6a29c5dee1fcdb7772c2"}
Oct 29 22:52:07.257864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.257653    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f38214cc445b6dce1ea09fd65bda3044aae55919c8bd6a29c5dee1fcdb7772c2"
Oct 29 22:52:07.261693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.261652    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" event={"ID":"8f5e2cbd-6e8f-47e0-a282-743b6525c169","Type":"ContainerStarted","Data":"70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2"}
Oct 29 22:52:07.261849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.261700    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" event={"ID":"8f5e2cbd-6e8f-47e0-a282-743b6525c169","Type":"ContainerStarted","Data":"8de9d4b45c117c89bd01ce6c33ab0971460cb1e0094c91d586ea7a186ea8e005"}
Oct 29 22:52:07.305797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.304001    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podStartSLOduration=1.303982935 podStartE2EDuration="1.303982935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:07.303133262 +0000 UTC m=+562.926044283" watchObservedRunningTime="2026-10-29 22:52:07.303982935 +0000 UTC m=+562.926893956"
Oct 29 22:52:07.799118 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:07.799084    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:15.670184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.670149    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"]
Oct 29 22:52:15.674200 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.674183    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.691228 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.691206    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:52:15.703658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.703625    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:52:15.705297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.705271    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"]
Oct 29 22:52:15.788290 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.788256    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.788497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.788432    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrvd\" (UniqueName: \"kubernetes.io/projected/aa11da47-5ad8-4944-99bf-f20bd8cadd10-kube-api-access-fgrvd\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.788497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.788494    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.790385 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.790364    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"]
Oct 29 22:52:15.794042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.794028    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.814074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.814051    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"]
Oct 29 22:52:15.889232 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.889205    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrvd\" (UniqueName: \"kubernetes.io/projected/aa11da47-5ad8-4944-99bf-f20bd8cadd10-kube-api-access-fgrvd\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.889383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.889248    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.889383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.889286    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqswv\" (UniqueName: \"kubernetes.io/projected/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-kube-api-access-vqswv\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.889383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.889314    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.889383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.889373    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.889601 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.889406    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.890077 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.890054    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.891745 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.891725    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.911844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.911822    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrvd\" (UniqueName: \"kubernetes.io/projected/aa11da47-5ad8-4944-99bf-f20bd8cadd10-kube-api-access-fgrvd\") pod \"maas-api-revoke-sub-5e815205-dl6lx\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.987987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.987918    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:15.990391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.990368    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqswv\" (UniqueName: \"kubernetes.io/projected/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-kube-api-access-vqswv\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.990513 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.990404    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.990513 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.990429    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.991262 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.991043    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:15.992843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:15.992817    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:16.013437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:16.013412    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqswv\" (UniqueName: \"kubernetes.io/projected/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-kube-api-access-vqswv\") pod \"maas-api-revoke-sub-580d6ee8-744rd\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:16.105752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:16.105717    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:16.284119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:16.284095    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"]
Oct 29 22:52:16.285638 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:16.285613    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa11da47_5ad8_4944_99bf_f20bd8cadd10.slice/crio-b02e4882c68a1f6cb630efce9362d62e8f0981c3c69f337b8acea5de897061c5 WatchSource:0}: Error finding container b02e4882c68a1f6cb630efce9362d62e8f0981c3c69f337b8acea5de897061c5: Status 404 returned error can't find the container with id b02e4882c68a1f6cb630efce9362d62e8f0981c3c69f337b8acea5de897061c5
Oct 29 22:52:16.308522 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:16.308499    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx" event={"ID":"aa11da47-5ad8-4944-99bf-f20bd8cadd10","Type":"ContainerStarted","Data":"b02e4882c68a1f6cb630efce9362d62e8f0981c3c69f337b8acea5de897061c5"}
Oct 29 22:52:16.405101 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:16.405071    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"]
Oct 29 22:52:16.406790 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:16.406747    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac43fdf9_f124_40c2_abc0_7f08ee6c60d7.slice/crio-789d73e455ed4be4f0a1c57994505eccb4d4d47300b72227576708c004909deb WatchSource:0}: Error finding container 789d73e455ed4be4f0a1c57994505eccb4d4d47300b72227576708c004909deb: Status 404 returned error can't find the container with id 789d73e455ed4be4f0a1c57994505eccb4d4d47300b72227576708c004909deb
Oct 29 22:52:16.799002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:16.798972    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:16.811489 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:16.811465    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:17.315419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:17.315330    4827 generic.go:356] "Generic (PLEG): container finished" podID="ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" containerID="97651af2869f98458c739f055782249ceed753a7bd3f16d9c943be64797341e8" exitCode=0
Oct 29 22:52:17.315582 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:17.315414    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd" event={"ID":"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7","Type":"ContainerDied","Data":"97651af2869f98458c739f055782249ceed753a7bd3f16d9c943be64797341e8"}
Oct 29 22:52:17.315582 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:17.315475    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd" event={"ID":"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7","Type":"ContainerStarted","Data":"789d73e455ed4be4f0a1c57994505eccb4d4d47300b72227576708c004909deb"}
Oct 29 22:52:17.317038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:17.317009    4827 generic.go:356] "Generic (PLEG): container finished" podID="aa11da47-5ad8-4944-99bf-f20bd8cadd10" containerID="64a31dfe3cd444c74d1eb7b17d38d6911af73beb05972996773de4471834334c" exitCode=0
Oct 29 22:52:17.317164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:17.317070    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx" event={"ID":"aa11da47-5ad8-4944-99bf-f20bd8cadd10","Type":"ContainerDied","Data":"64a31dfe3cd444c74d1eb7b17d38d6911af73beb05972996773de4471834334c"}
Oct 29 22:52:17.328404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:17.328382    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:52:18.482431 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.482163    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:18.516109 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.515757    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:18.617720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.617646    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-openshift-ca\") pod \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") "
Oct 29 22:52:18.617885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.617719    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-openshift-ca\") pod \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") "
Oct 29 22:52:18.617885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.617748    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-kube-api-access-vqswv\" (UniqueName: \"kubernetes.io/projected/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-kube-api-access-vqswv\") pod \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") "
Oct 29 22:52:18.617885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.617799    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa11da47-5ad8-4944-99bf-f20bd8cadd10-kube-api-access-fgrvd\" (UniqueName: \"kubernetes.io/projected/aa11da47-5ad8-4944-99bf-f20bd8cadd10-kube-api-access-fgrvd\") pod \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") "
Oct 29 22:52:18.618018 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.617909    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-xks-ca\") pod \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\" (UID: \"aa11da47-5ad8-4944-99bf-f20bd8cadd10\") "
Oct 29 22:52:18.618075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.618024    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-xks-ca\") pod \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\" (UID: \"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7\") "
Oct 29 22:52:18.618129 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.618102    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-openshift-ca" pod "aa11da47-5ad8-4944-99bf-f20bd8cadd10" (UID: "aa11da47-5ad8-4944-99bf-f20bd8cadd10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:18.618194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.618106    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-openshift-ca" pod "ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" (UID: "ac43fdf9-f124-40c2-abc0-7f08ee6c60d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:18.618551 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.618506    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:18.618551 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.618531    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:18.620114 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.620085    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-xks-ca" pod "aa11da47-5ad8-4944-99bf-f20bd8cadd10" (UID: "aa11da47-5ad8-4944-99bf-f20bd8cadd10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:18.620200 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.620162    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-xks-ca" pod "ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" (UID: "ac43fdf9-f124-40c2-abc0-7f08ee6c60d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:18.620428 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.620405    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa11da47-5ad8-4944-99bf-f20bd8cadd10-kube-api-access-fgrvd" pod "aa11da47-5ad8-4944-99bf-f20bd8cadd10" (UID: "aa11da47-5ad8-4944-99bf-f20bd8cadd10"). InnerVolumeSpecName "kube-api-access-fgrvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:18.620517 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.620430    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-kube-api-access-vqswv" pod "ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" (UID: "ac43fdf9-f124-40c2-abc0-7f08ee6c60d7"). InnerVolumeSpecName "kube-api-access-vqswv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:18.719876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.719850    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqswv\" (UniqueName: \"kubernetes.io/projected/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-kube-api-access-vqswv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:18.719876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.719871    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrvd\" (UniqueName: \"kubernetes.io/projected/aa11da47-5ad8-4944-99bf-f20bd8cadd10-kube-api-access-fgrvd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:18.720047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.719883    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11da47-5ad8-4944-99bf-f20bd8cadd10-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:18.720047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:18.719895    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:19.330075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:19.330051    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"
Oct 29 22:52:19.330240 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:19.330091    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd" event={"ID":"ac43fdf9-f124-40c2-abc0-7f08ee6c60d7","Type":"ContainerDied","Data":"789d73e455ed4be4f0a1c57994505eccb4d4d47300b72227576708c004909deb"}
Oct 29 22:52:19.330240 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:19.330173    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789d73e455ed4be4f0a1c57994505eccb4d4d47300b72227576708c004909deb"
Oct 29 22:52:19.331635 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:19.331600    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx" event={"ID":"aa11da47-5ad8-4944-99bf-f20bd8cadd10","Type":"ContainerDied","Data":"b02e4882c68a1f6cb630efce9362d62e8f0981c3c69f337b8acea5de897061c5"}
Oct 29 22:52:19.331635 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:19.331613    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"
Oct 29 22:52:19.331846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:19.331655    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02e4882c68a1f6cb630efce9362d62e8f0981c3c69f337b8acea5de897061c5"
Oct 29 22:52:26.607111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.607078    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"]
Oct 29 22:52:26.607498 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.607487    4827 cpu_manager.go:414] "containerMap: removing container" podUID="aa11da47-5ad8-4944-99bf-f20bd8cadd10" containerName="revoke-keys"
Oct 29 22:52:26.607546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.607499    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa11da47-5ad8-4944-99bf-f20bd8cadd10" containerName="revoke-keys"
Oct 29 22:52:26.607546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.607524    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" containerName="revoke-keys"
Oct 29 22:52:26.607546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.607530    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" containerName="revoke-keys"
Oct 29 22:52:26.607642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.607600    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" containerName="revoke-keys"
Oct 29 22:52:26.607642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.607610    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa11da47-5ad8-4944-99bf-f20bd8cadd10" containerName="revoke-keys"
Oct 29 22:52:26.611961 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.611945    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.628478 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.628447    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:52:26.649826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.649756    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:52:26.654561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.654526    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"]
Oct 29 22:52:26.717302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.717272    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.717445 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.717410    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.717483 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.717450    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cls29\" (UniqueName: \"kubernetes.io/projected/8f682e56-8ac0-4383-b884-2a84f47c07d2-kube-api-access-cls29\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.818852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.818823    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.819038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.818892    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.819038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.818920    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cls29\" (UniqueName: \"kubernetes.io/projected/8f682e56-8ac0-4383-b884-2a84f47c07d2-kube-api-access-cls29\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.819532 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.819508    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.821234 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.821211    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.842154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.842132    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cls29\" (UniqueName: \"kubernetes.io/projected/8f682e56-8ac0-4383-b884-2a84f47c07d2-kube-api-access-cls29\") pod \"maas-api-revoke-sub-b5879ada-69qm7\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:26.925316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:26.925255    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:27.241356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:27.241327    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"]
Oct 29 22:52:27.242941 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:27.242920    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f682e56_8ac0_4383_b884_2a84f47c07d2.slice/crio-ce647384e7637e7d602988e8fa8213dcc38792531cdf1ccf9a088b855310e3e4 WatchSource:0}: Error finding container ce647384e7637e7d602988e8fa8213dcc38792531cdf1ccf9a088b855310e3e4: Status 404 returned error can't find the container with id ce647384e7637e7d602988e8fa8213dcc38792531cdf1ccf9a088b855310e3e4
Oct 29 22:52:27.388465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:27.388407    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7" event={"ID":"8f682e56-8ac0-4383-b884-2a84f47c07d2","Type":"ContainerStarted","Data":"ce647384e7637e7d602988e8fa8213dcc38792531cdf1ccf9a088b855310e3e4"}
Oct 29 22:52:28.395816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.395785    4827 generic.go:356] "Generic (PLEG): container finished" podID="8f682e56-8ac0-4383-b884-2a84f47c07d2" containerID="96b864d94ae9e4588b2459a0a67d4cd5799e351b4cbf37dcdc116e5dd583a064" exitCode=0
Oct 29 22:52:28.396244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.395822    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7" event={"ID":"8f682e56-8ac0-4383-b884-2a84f47c07d2","Type":"ContainerDied","Data":"96b864d94ae9e4588b2459a0a67d4cd5799e351b4cbf37dcdc116e5dd583a064"}
Oct 29 22:52:28.751695 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.751604    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"]
Oct 29 22:52:28.758848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.757487    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.778373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.778339    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"]
Oct 29 22:52:28.784842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.784818    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"openshift-service-ca.crt\""
Oct 29 22:52:28.784842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.784846    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"kube-root-ca.crt\""
Oct 29 22:52:28.785406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.785390    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"default-dockercfg-mtks9\""
Oct 29 22:52:28.785534 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.785421    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"premium-w3-17394a-kserve-self-signed-certs\""
Oct 29 22:52:28.841720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.841691    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-model-cache\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.841896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.841732    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75t4n\" (UniqueName: \"kubernetes.io/projected/2e711f2b-f138-4812-adc1-07e9dd15aadd-kube-api-access-75t4n\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.841896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.841820    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-home\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.841896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.841850    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-tmp-dir\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.842004 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.841902    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-dshm\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.842037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.842002    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e711f2b-f138-4812-adc1-07e9dd15aadd-tls-certs\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.943447 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.943416    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-dshm\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.943640 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.943460    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e711f2b-f138-4812-adc1-07e9dd15aadd-tls-certs\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.943640 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.943547    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-model-cache\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.943640 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.943587    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75t4n\" (UniqueName: \"kubernetes.io/projected/2e711f2b-f138-4812-adc1-07e9dd15aadd-kube-api-access-75t4n\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.943840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.943642    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-home\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.943840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.943666    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-tmp-dir\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.943980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.943951    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-model-cache\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.944044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.944000    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-tmp-dir\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.944084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.944049    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-home\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.945826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.945805    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-dshm\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.945982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.945967    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e711f2b-f138-4812-adc1-07e9dd15aadd-tls-certs\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:28.994505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:28.994456    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75t4n\" (UniqueName: \"kubernetes.io/projected/2e711f2b-f138-4812-adc1-07e9dd15aadd-kube-api-access-75t4n\") pod \"premium-w3-17394a-kserve-855c947754-2hhkg\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") " pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:29.073126 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.073052    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:29.369750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.369725    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"]
Oct 29 22:52:29.371868 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:29.371832    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e711f2b_f138_4812_adc1_07e9dd15aadd.slice/crio-8470b1edb5044b3c92d348e156e6be28a6ccd99fa5da23f07b5bfdb612c5a2fc WatchSource:0}: Error finding container 8470b1edb5044b3c92d348e156e6be28a6ccd99fa5da23f07b5bfdb612c5a2fc: Status 404 returned error can't find the container with id 8470b1edb5044b3c92d348e156e6be28a6ccd99fa5da23f07b5bfdb612c5a2fc
Oct 29 22:52:29.402543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.402513    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" event={"ID":"2e711f2b-f138-4812-adc1-07e9dd15aadd","Type":"ContainerStarted","Data":"8470b1edb5044b3c92d348e156e6be28a6ccd99fa5da23f07b5bfdb612c5a2fc"}
Oct 29 22:52:29.482958 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.482413    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"]
Oct 29 22:52:29.487397 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.486924    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.513865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.513835    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"]
Oct 29 22:52:29.543292 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.543266    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:29.549635 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549611    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.549752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549646    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.549752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549689    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.549752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549747    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.549902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549835    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.549902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549858    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.549902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549880    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.550020 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549965    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8469\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-kube-api-access-s8469\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.550020 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.549992    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651021 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.650993    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-openshift-ca\") pod \"8f682e56-8ac0-4383-b884-2a84f47c07d2\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") "
Oct 29 22:52:29.651182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651060    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f682e56-8ac0-4383-b884-2a84f47c07d2-kube-api-access-cls29\" (UniqueName: \"kubernetes.io/projected/8f682e56-8ac0-4383-b884-2a84f47c07d2-kube-api-access-cls29\") pod \"8f682e56-8ac0-4383-b884-2a84f47c07d2\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") "
Oct 29 22:52:29.651226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651208    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-xks-ca\") pod \"8f682e56-8ac0-4383-b884-2a84f47c07d2\" (UID: \"8f682e56-8ac0-4383-b884-2a84f47c07d2\") "
Oct 29 22:52:29.651373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651354    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-openshift-ca" pod "8f682e56-8ac0-4383-b884-2a84f47c07d2" (UID: "8f682e56-8ac0-4383-b884-2a84f47c07d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:29.651448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651426    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651530 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651501    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651590 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651539    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651590 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651578    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651674    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8469\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-kube-api-access-s8469\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651736    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651814    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651850    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.651965 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651943    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.652079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.651965    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.652079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.652025    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:29.652647 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.652620    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.652797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.652725    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.652968 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.652940    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.653124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.653103    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.653493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.653460    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-xks-ca" pod "8f682e56-8ac0-4383-b884-2a84f47c07d2" (UID: "8f682e56-8ac0-4383-b884-2a84f47c07d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:29.653589 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.653523    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f682e56-8ac0-4383-b884-2a84f47c07d2-kube-api-access-cls29" pod "8f682e56-8ac0-4383-b884-2a84f47c07d2" (UID: "8f682e56-8ac0-4383-b884-2a84f47c07d2"). InnerVolumeSpecName "kube-api-access-cls29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:29.654133 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.654111    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.655116 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.655096    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.675085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.675064    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.675187 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.675125    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8469\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-kube-api-access-s8469\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:29.756782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.754133    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cls29\" (UniqueName: \"kubernetes.io/projected/8f682e56-8ac0-4383-b884-2a84f47c07d2-kube-api-access-cls29\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:29.756782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.754173    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f682e56-8ac0-4383-b884-2a84f47c07d2-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:52:29.806528 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:29.806492    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:30.129423 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.129396    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"]
Oct 29 22:52:30.131050 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:30.131019    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5f9825_0f08_40ce_a49d_d8039190f8e3.slice/crio-c7b7f67dc9119bf78442543536df362abfd640a1b108a4948cb7c4a5ab8944d2 WatchSource:0}: Error finding container c7b7f67dc9119bf78442543536df362abfd640a1b108a4948cb7c4a5ab8944d2: Status 404 returned error can't find the container with id c7b7f67dc9119bf78442543536df362abfd640a1b108a4948cb7c4a5ab8944d2
Oct 29 22:52:30.133624 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.133589    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:52:30.133725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.133657    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:52:30.133725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.133687    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:52:30.411440 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.411389    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"
Oct 29 22:52:30.411919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.411392    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7" event={"ID":"8f682e56-8ac0-4383-b884-2a84f47c07d2","Type":"ContainerDied","Data":"ce647384e7637e7d602988e8fa8213dcc38792531cdf1ccf9a088b855310e3e4"}
Oct 29 22:52:30.411919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.411571    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce647384e7637e7d602988e8fa8213dcc38792531cdf1ccf9a088b855310e3e4"
Oct 29 22:52:30.413211 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.413187    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" event={"ID":"2e711f2b-f138-4812-adc1-07e9dd15aadd","Type":"ContainerStarted","Data":"72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774"}
Oct 29 22:52:30.422293 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.422265    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" event={"ID":"2c5f9825-0f08-40ce-a49d-d8039190f8e3","Type":"ContainerStarted","Data":"44c04dd6445d22cda20fd0799a10a8e5ff322d323f014a781351bd0b70f5044d"}
Oct 29 22:52:30.422422 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.422321    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" event={"ID":"2c5f9825-0f08-40ce-a49d-d8039190f8e3","Type":"ContainerStarted","Data":"c7b7f67dc9119bf78442543536df362abfd640a1b108a4948cb7c4a5ab8944d2"}
Oct 29 22:52:30.455717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.455673    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podStartSLOduration=2.455657059 podStartE2EDuration="2.455657059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:30.454971733 +0000 UTC m=+586.077882753" watchObservedRunningTime="2026-10-29 22:52:30.455657059 +0000 UTC m=+586.078568081"
Oct 29 22:52:30.515275 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.515221    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podStartSLOduration=1.515205356 podStartE2EDuration="1.515205356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:30.513218481 +0000 UTC m=+586.136129502" watchObservedRunningTime="2026-10-29 22:52:30.515205356 +0000 UTC m=+586.138116376"
Oct 29 22:52:30.806833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:30.806718    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:31.076734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:31.076609    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:31.808613 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:31.808580    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:52:31.809033 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:31.808632    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:52:32.809792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:32.808241    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:52:32.809792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:32.808318    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:52:32.857288 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:52:32.857260    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:52:33.807467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:33.807427    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:52:33.807649 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:33.807489    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:52:34.806943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:34.806913    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:52:34.807414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:34.806968    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:52:35.807432 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:35.807396    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:52:35.807882 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:35.807450    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:52:37.705481 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:37.705433    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:37.706584 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:37.706523    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:37.707698 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:37.707672    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 22:52:39.074166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:39.074130    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:39.087038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:39.087017    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:39.484604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:39.484573    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:52:42.489827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.489796    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"]
Oct 29 22:52:42.490280 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.490263    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8f682e56-8ac0-4383-b884-2a84f47c07d2" containerName="revoke-keys"
Oct 29 22:52:42.490339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.490282    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f682e56-8ac0-4383-b884-2a84f47c07d2" containerName="revoke-keys"
Oct 29 22:52:42.490448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.490420    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f682e56-8ac0-4383-b884-2a84f47c07d2" containerName="revoke-keys"
Oct 29 22:52:42.494988 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.494968    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.513638 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.513614    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"]
Oct 29 22:52:42.521982 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.521960    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"premium-w2-575d59-kserve-self-signed-certs\""
Oct 29 22:52:42.635011 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.634968    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm8z7\" (UniqueName: \"kubernetes.io/projected/2081f6b2-15ff-4280-9d56-917613a6af5e-kube-api-access-hm8z7\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.635011 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.635015    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-tmp-dir\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.635219 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.635163    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2081f6b2-15ff-4280-9d56-917613a6af5e-tls-certs\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.635219 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.635204    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-dshm\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.635313 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.635295    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-home\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.635377 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.635364    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-model-cache\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.736792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.736195    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-home\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.736792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.736332    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-model-cache\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.736792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.736434    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm8z7\" (UniqueName: \"kubernetes.io/projected/2081f6b2-15ff-4280-9d56-917613a6af5e-kube-api-access-hm8z7\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.736792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.736478    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-tmp-dir\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.736792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.736551    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2081f6b2-15ff-4280-9d56-917613a6af5e-tls-certs\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.736792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.736571    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-dshm\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.737722 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.737695    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-home\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.738872 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.738139    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-tmp-dir\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.739027 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.738356    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-model-cache\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.757661 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.757583    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-dshm\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.761063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.761026    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2081f6b2-15ff-4280-9d56-917613a6af5e-tls-certs\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.761282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.761104    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm8z7\" (UniqueName: \"kubernetes.io/projected/2081f6b2-15ff-4280-9d56-917613a6af5e-kube-api-access-hm8z7\") pod \"premium-w2-575d59-kserve-7bdbc84585-d9d7s\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") " pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:42.810325 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:42.810290    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:43.109322 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:43.109298    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"]
Oct 29 22:52:43.110940 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:43.110909    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2081f6b2_15ff_4280_9d56_917613a6af5e.slice/crio-0d5989529f400825ee3df154c95c9f87c548242696359f36faf392b5ed915e07 WatchSource:0}: Error finding container 0d5989529f400825ee3df154c95c9f87c548242696359f36faf392b5ed915e07: Status 404 returned error can't find the container with id 0d5989529f400825ee3df154c95c9f87c548242696359f36faf392b5ed915e07
Oct 29 22:52:43.497423 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:43.497384    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" event={"ID":"2081f6b2-15ff-4280-9d56-917613a6af5e","Type":"ContainerStarted","Data":"507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649"}
Oct 29 22:52:43.497423 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:43.497428    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" event={"ID":"2081f6b2-15ff-4280-9d56-917613a6af5e","Type":"ContainerStarted","Data":"0d5989529f400825ee3df154c95c9f87c548242696359f36faf392b5ed915e07"}
Oct 29 22:52:43.539782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:43.539723    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podStartSLOduration=1.5397095159999998 podStartE2EDuration="1.539709516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:43.537730262 +0000 UTC m=+599.160641282" watchObservedRunningTime="2026-10-29 22:52:43.539709516 +0000 UTC m=+599.162620536"
Oct 29 22:52:43.810581 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:43.810504    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:44.559256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:44.559227    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:52:44.559893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:44.559449    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:52:47.598551 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.598519    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"]
Oct 29 22:52:47.603226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.603204    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.630694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.630648    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"default-dockercfg-b7z2q\""
Oct 29 22:52:47.630694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.630665    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"kube-root-ca.crt\""
Oct 29 22:52:47.630694 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.630674    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"openshift-service-ca.crt\""
Oct 29 22:52:47.630957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.630704    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"premium-w5-b2921b-kserve-self-signed-certs\""
Oct 29 22:52:47.633423 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.633398    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"]
Oct 29 22:52:47.725327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.725297    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-model-cache\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.725485 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.725366    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-dshm\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.725485 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.725427    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-home\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.725614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.725486    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmn9\" (UniqueName: \"kubernetes.io/projected/94b989aa-8e73-42af-954c-76fbd2148199-kube-api-access-skmn9\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.725676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.725650    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-tmp-dir\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.725728 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.725691    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b989aa-8e73-42af-954c-76fbd2148199-tls-certs\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.832134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.832095    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-model-cache\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.832413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.832394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-dshm\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.832535 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.832520    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-home\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.832709 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.832693    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmn9\" (UniqueName: \"kubernetes.io/projected/94b989aa-8e73-42af-954c-76fbd2148199-kube-api-access-skmn9\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.832911 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.832896    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-tmp-dir\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.833013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.833001    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b989aa-8e73-42af-954c-76fbd2148199-tls-certs\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.833159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.832732    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-model-cache\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.833266 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.833059    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-home\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.833387 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.833266    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-tmp-dir\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.853524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.853452    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-dshm\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.855427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.855368    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b989aa-8e73-42af-954c-76fbd2148199-tls-certs\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.859019 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.858989    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmn9\" (UniqueName: \"kubernetes.io/projected/94b989aa-8e73-42af-954c-76fbd2148199-kube-api-access-skmn9\") pod \"premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") " pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:47.918391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:47.918361    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:48.224206 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:48.224183    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"]
Oct 29 22:52:48.226549 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:52:48.226518    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94b989aa_8e73_42af_954c_76fbd2148199.slice/crio-d560a512850800f30ac57b464797202ce4e6ddd032b0067b8ea9debbb125feb6 WatchSource:0}: Error finding container d560a512850800f30ac57b464797202ce4e6ddd032b0067b8ea9debbb125feb6: Status 404 returned error can't find the container with id d560a512850800f30ac57b464797202ce4e6ddd032b0067b8ea9debbb125feb6
Oct 29 22:52:48.529802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:48.529705    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" event={"ID":"94b989aa-8e73-42af-954c-76fbd2148199","Type":"ContainerStarted","Data":"52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37"}
Oct 29 22:52:48.529802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:48.529742    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" event={"ID":"94b989aa-8e73-42af-954c-76fbd2148199","Type":"ContainerStarted","Data":"d560a512850800f30ac57b464797202ce4e6ddd032b0067b8ea9debbb125feb6"}
Oct 29 22:52:48.572867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:48.572809    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podStartSLOduration=1.572790328 podStartE2EDuration="1.572790328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:48.569748348 +0000 UTC m=+604.192659368" watchObservedRunningTime="2026-10-29 22:52:48.572790328 +0000 UTC m=+604.195701351"
Oct 29 22:52:48.918997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:48.918969    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:52.811011 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:52.810978    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:52.823762 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:52.823736    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:53.567063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:53.567035    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:52:57.931847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:57.931818    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:57.997228 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:57.997200    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:52:58.592609 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:52:58.592583    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:53:02.822832 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:53:02.822798    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:53:04.081480 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.081446    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"]
Oct 29 22:53:04.085332 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.085316    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.102283 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.102258    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:53:04.114237 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.114213    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:53:04.117245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.117223    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"]
Oct 29 22:53:04.163981 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.163950    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgp2c\" (UniqueName: \"kubernetes.io/projected/19792e88-e399-4e25-aa41-79f802825e55-kube-api-access-cgp2c\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.164124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.164073    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19792e88-e399-4e25-aa41-79f802825e55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.164124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.164108    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19792e88-e399-4e25-aa41-79f802825e55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.265174 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.265141    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19792e88-e399-4e25-aa41-79f802825e55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.265335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.265186    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19792e88-e399-4e25-aa41-79f802825e55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.265389 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.265352    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgp2c\" (UniqueName: \"kubernetes.io/projected/19792e88-e399-4e25-aa41-79f802825e55-kube-api-access-cgp2c\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.265761 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.265741    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19792e88-e399-4e25-aa41-79f802825e55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.267476 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.267452    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19792e88-e399-4e25-aa41-79f802825e55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.289437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.289418    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgp2c\" (UniqueName: \"kubernetes.io/projected/19792e88-e399-4e25-aa41-79f802825e55-kube-api-access-cgp2c\") pod \"maas-api-revoke-sub-23da404e-fswcw\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.399067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.399033    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:04.695592 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:04.695568    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"]
Oct 29 22:53:04.697813 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:53:04.697788    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19792e88_e399_4e25_aa41_79f802825e55.slice/crio-6074571d49fbf0a93f6d8af0833f132389b398c31b020498753e52b7800b2e9c WatchSource:0}: Error finding container 6074571d49fbf0a93f6d8af0833f132389b398c31b020498753e52b7800b2e9c: Status 404 returned error can't find the container with id 6074571d49fbf0a93f6d8af0833f132389b398c31b020498753e52b7800b2e9c
Oct 29 22:53:05.648501 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:05.648467    4827 generic.go:356] "Generic (PLEG): container finished" podID="19792e88-e399-4e25-aa41-79f802825e55" containerID="e05ddf44ef8b1faea72dd1533c512e1b92c94f0236c23af7e674633f71b875f9" exitCode=0
Oct 29 22:53:05.648904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:05.648553    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw" event={"ID":"19792e88-e399-4e25-aa41-79f802825e55","Type":"ContainerDied","Data":"e05ddf44ef8b1faea72dd1533c512e1b92c94f0236c23af7e674633f71b875f9"}
Oct 29 22:53:05.648904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:05.648612    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw" event={"ID":"19792e88-e399-4e25-aa41-79f802825e55","Type":"ContainerStarted","Data":"6074571d49fbf0a93f6d8af0833f132389b398c31b020498753e52b7800b2e9c"}
Oct 29 22:53:06.799443 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.799417    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:06.891168 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.891137    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19792e88-e399-4e25-aa41-79f802825e55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19792e88-e399-4e25-aa41-79f802825e55-maas-api-openshift-ca\") pod \"19792e88-e399-4e25-aa41-79f802825e55\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") "
Oct 29 22:53:06.891359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.891211    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19792e88-e399-4e25-aa41-79f802825e55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19792e88-e399-4e25-aa41-79f802825e55-maas-api-xks-ca\") pod \"19792e88-e399-4e25-aa41-79f802825e55\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") "
Oct 29 22:53:06.891359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.891337    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19792e88-e399-4e25-aa41-79f802825e55-kube-api-access-cgp2c\" (UniqueName: \"kubernetes.io/projected/19792e88-e399-4e25-aa41-79f802825e55-kube-api-access-cgp2c\") pod \"19792e88-e399-4e25-aa41-79f802825e55\" (UID: \"19792e88-e399-4e25-aa41-79f802825e55\") "
Oct 29 22:53:06.891531 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.891507    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19792e88-e399-4e25-aa41-79f802825e55-maas-api-openshift-ca" pod "19792e88-e399-4e25-aa41-79f802825e55" (UID: "19792e88-e399-4e25-aa41-79f802825e55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:06.891799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.891753    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19792e88-e399-4e25-aa41-79f802825e55-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:53:06.893296 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.893271    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19792e88-e399-4e25-aa41-79f802825e55-kube-api-access-cgp2c" pod "19792e88-e399-4e25-aa41-79f802825e55" (UID: "19792e88-e399-4e25-aa41-79f802825e55"). InnerVolumeSpecName "kube-api-access-cgp2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:06.893396 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.893369    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19792e88-e399-4e25-aa41-79f802825e55-maas-api-xks-ca" pod "19792e88-e399-4e25-aa41-79f802825e55" (UID: "19792e88-e399-4e25-aa41-79f802825e55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:06.992880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.992800    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgp2c\" (UniqueName: \"kubernetes.io/projected/19792e88-e399-4e25-aa41-79f802825e55-kube-api-access-cgp2c\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:53:06.992880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:06.992826    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19792e88-e399-4e25-aa41-79f802825e55-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:53:07.661131 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:07.661101    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"
Oct 29 22:53:07.661131 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:07.661117    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw" event={"ID":"19792e88-e399-4e25-aa41-79f802825e55","Type":"ContainerDied","Data":"6074571d49fbf0a93f6d8af0833f132389b398c31b020498753e52b7800b2e9c"}
Oct 29 22:53:07.661355 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:07.661160    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6074571d49fbf0a93f6d8af0833f132389b398c31b020498753e52b7800b2e9c"
Oct 29 22:53:18.586454 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.586418    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"]
Oct 29 22:53:18.587511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.587484    4827 cpu_manager.go:414] "containerMap: removing container" podUID="19792e88-e399-4e25-aa41-79f802825e55" containerName="revoke-keys"
Oct 29 22:53:18.587671 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.587654    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19792e88-e399-4e25-aa41-79f802825e55" containerName="revoke-keys"
Oct 29 22:53:18.588023 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.588000    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="19792e88-e399-4e25-aa41-79f802825e55" containerName="revoke-keys"
Oct 29 22:53:18.593933 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.593908    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.608010 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.607988    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"]
Oct 29 22:53:18.617400 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.617376    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"distinct-w4-063b46-kserve-self-signed-certs\""
Oct 29 22:53:18.652406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.652380    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-model-cache\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.652541 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.652416    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tmp-dir\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.652541 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.652530    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tls-certs\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.652624 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.652608    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2745b\" (UniqueName: \"kubernetes.io/projected/712e715a-7d5a-41df-a6a9-7d04f90c07b8-kube-api-access-2745b\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.652697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.652675    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-dshm\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.652737 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.652721    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-home\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.753401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.753371    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-dshm\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.753587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.753424    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-home\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.753587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.753511    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-model-cache\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.753587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.753530    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tmp-dir\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.753587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.753567    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tls-certs\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.753851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.753628    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2745b\" (UniqueName: \"kubernetes.io/projected/712e715a-7d5a-41df-a6a9-7d04f90c07b8-kube-api-access-2745b\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.754182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.754155    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-home\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.754307 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.754188    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tmp-dir\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.754372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.754319    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-model-cache\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.756178 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.756133    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-dshm\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.758726 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.756687    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tls-certs\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.797736 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.797703    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2745b\" (UniqueName: \"kubernetes.io/projected/712e715a-7d5a-41df-a6a9-7d04f90c07b8-kube-api-access-2745b\") pod \"distinct-w4-063b46-kserve-78b64b9fd8-dtrf8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:18.907110 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:18.907078    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:19.211130 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:19.211101    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"]
Oct 29 22:53:19.212484 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:53:19.212457    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod712e715a_7d5a_41df_a6a9_7d04f90c07b8.slice/crio-4d70dc3035503f7b075b7b6aa5db485a95fd323e095f87299612933d087b177d WatchSource:0}: Error finding container 4d70dc3035503f7b075b7b6aa5db485a95fd323e095f87299612933d087b177d: Status 404 returned error can't find the container with id 4d70dc3035503f7b075b7b6aa5db485a95fd323e095f87299612933d087b177d
Oct 29 22:53:19.748755 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:19.748722    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" event={"ID":"712e715a-7d5a-41df-a6a9-7d04f90c07b8","Type":"ContainerStarted","Data":"7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca"}
Oct 29 22:53:19.748755 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:19.748776    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" event={"ID":"712e715a-7d5a-41df-a6a9-7d04f90c07b8","Type":"ContainerStarted","Data":"4d70dc3035503f7b075b7b6aa5db485a95fd323e095f87299612933d087b177d"}
Oct 29 22:53:19.790663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:19.790610    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podStartSLOduration=1.7905947709999999 podStartE2EDuration="1.790594771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:19.789210613 +0000 UTC m=+635.412121634" watchObservedRunningTime="2026-10-29 22:53:19.790594771 +0000 UTC m=+635.413505790"
Oct 29 22:53:19.909901 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:19.909866    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:28.907543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:28.907509    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:28.922175 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:28.922154    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:29.666461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.666427    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"]
Oct 29 22:53:29.670259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.670237    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.700850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.700815    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"]
Oct 29 22:53:29.724313 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724283    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724360    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724422    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724607 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724545    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724607 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724595    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724678 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724624    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4fb03b-5491-4d64-a680-2ab271a593b5-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724712 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724689    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtm8\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-kube-api-access-hmtm8\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724710    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.724809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.724730    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826121 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826082    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826140    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826192    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826214    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826236    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4fb03b-5491-4d64-a680-2ab271a593b5-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826277    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtm8\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-kube-api-access-hmtm8\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826304    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826324    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826528    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.826802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.826780    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.827071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.827050    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.827184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.827161    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4fb03b-5491-4d64-a680-2ab271a593b5-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.827248 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.827171    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.828995 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.828974    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.829100 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.829078    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.840646 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.840623    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:53:29.849698 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.849672    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.849806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.849753    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtm8\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-kube-api-access-hmtm8\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:29.986085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:29.985987    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:30.314271 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.314245    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"]
Oct 29 22:53:30.315684 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:53:30.315651    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4fb03b_5491_4d64_a680_2ab271a593b5.slice/crio-5de91f0f19e9ffdabdcf272a22fa7780a8024546fa3362c42a32a949c7db8560 WatchSource:0}: Error finding container 5de91f0f19e9ffdabdcf272a22fa7780a8024546fa3362c42a32a949c7db8560: Status 404 returned error can't find the container with id 5de91f0f19e9ffdabdcf272a22fa7780a8024546fa3362c42a32a949c7db8560
Oct 29 22:53:30.318048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.318007    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:30.318138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.318091    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:30.318173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.318131    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:30.837026 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.836988    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" event={"ID":"7e4fb03b-5491-4d64-a680-2ab271a593b5","Type":"ContainerStarted","Data":"ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515"}
Oct 29 22:53:30.837223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.837049    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" event={"ID":"7e4fb03b-5491-4d64-a680-2ab271a593b5","Type":"ContainerStarted","Data":"5de91f0f19e9ffdabdcf272a22fa7780a8024546fa3362c42a32a949c7db8560"}
Oct 29 22:53:30.923525 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.923468    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podStartSLOduration=1.923455734 podStartE2EDuration="1.923455734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:30.91125451 +0000 UTC m=+646.534165530" watchObservedRunningTime="2026-10-29 22:53:30.923455734 +0000 UTC m=+646.546366754"
Oct 29 22:53:30.987905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:30.987867    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:31.989669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:31.989631    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:53:31.990112 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:31.989700    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:53:32.824735 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:53:32.824700    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:53:32.988372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:32.988336    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:53:32.988562 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:32.988404    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:53:33.851145 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.851107    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"]
Oct 29 22:53:33.855281 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.855261    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:33.885980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.885952    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:53:33.887307 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.886850    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:53:33.892377 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.892352    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"]
Oct 29 22:53:33.927002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.926975    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:33.927178 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.927111    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8dhv\" (UniqueName: \"kubernetes.io/projected/080b5369-dcb9-491c-968b-e209b8b6a807-kube-api-access-r8dhv\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:33.927257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.927177    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:33.988134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.988104    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:53:33.988329 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:33.988208    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:53:34.028188 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.028163    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:34.028375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.028255    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:34.028444 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.028430    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8dhv\" (UniqueName: \"kubernetes.io/projected/080b5369-dcb9-491c-968b-e209b8b6a807-kube-api-access-r8dhv\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:34.029014 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.028991    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:34.030493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.030475    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:34.053926 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.053900    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8dhv\" (UniqueName: \"kubernetes.io/projected/080b5369-dcb9-491c-968b-e209b8b6a807-kube-api-access-r8dhv\") pod \"maas-api-revoke-sub-688e0e11-rx2cw\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:34.192215 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.192005    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:34.529789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.519111    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"]
Oct 29 22:53:34.541709 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:53:34.541675    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080b5369_dcb9_491c_968b_e209b8b6a807.slice/crio-dbc12cab949e06fb5b69903435b2cd6573f4691928705b3cd2167201e4bbf95b WatchSource:0}: Error finding container dbc12cab949e06fb5b69903435b2cd6573f4691928705b3cd2167201e4bbf95b: Status 404 returned error can't find the container with id dbc12cab949e06fb5b69903435b2cd6573f4691928705b3cd2167201e4bbf95b
Oct 29 22:53:34.886790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.881299    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw" event={"ID":"080b5369-dcb9-491c-968b-e209b8b6a807","Type":"ContainerStarted","Data":"dbc12cab949e06fb5b69903435b2cd6573f4691928705b3cd2167201e4bbf95b"}
Oct 29 22:53:34.988262 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.988182    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:53:34.988262 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:34.988242    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:53:35.895066 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:35.895030    4827 generic.go:356] "Generic (PLEG): container finished" podID="080b5369-dcb9-491c-968b-e209b8b6a807" containerID="d0a9cee90ec6e65a7d0b7754dcb6d8e025681b73c6d61c02812a486ac23f07e9" exitCode=0
Oct 29 22:53:35.895526 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:35.895108    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw" event={"ID":"080b5369-dcb9-491c-968b-e209b8b6a807","Type":"ContainerDied","Data":"d0a9cee90ec6e65a7d0b7754dcb6d8e025681b73c6d61c02812a486ac23f07e9"}
Oct 29 22:53:35.988191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:35.988100    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:53:35.988191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:35.988155    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:53:37.070408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.070376    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:37.165338 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.165251    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080b5369-dcb9-491c-968b-e209b8b6a807-kube-api-access-r8dhv\" (UniqueName: \"kubernetes.io/projected/080b5369-dcb9-491c-968b-e209b8b6a807-kube-api-access-r8dhv\") pod \"080b5369-dcb9-491c-968b-e209b8b6a807\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") "
Oct 29 22:53:37.165338 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.165300    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-xks-ca\") pod \"080b5369-dcb9-491c-968b-e209b8b6a807\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") "
Oct 29 22:53:37.165631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.165433    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-openshift-ca\") pod \"080b5369-dcb9-491c-968b-e209b8b6a807\" (UID: \"080b5369-dcb9-491c-968b-e209b8b6a807\") "
Oct 29 22:53:37.165829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.165805    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-openshift-ca" pod "080b5369-dcb9-491c-968b-e209b8b6a807" (UID: "080b5369-dcb9-491c-968b-e209b8b6a807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:37.165990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.165936    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:53:37.167343 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.167315    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-xks-ca" pod "080b5369-dcb9-491c-968b-e209b8b6a807" (UID: "080b5369-dcb9-491c-968b-e209b8b6a807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:37.167865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.167841    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080b5369-dcb9-491c-968b-e209b8b6a807-kube-api-access-r8dhv" pod "080b5369-dcb9-491c-968b-e209b8b6a807" (UID: "080b5369-dcb9-491c-968b-e209b8b6a807"). InnerVolumeSpecName "kube-api-access-r8dhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:37.267210 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.267179    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8dhv\" (UniqueName: \"kubernetes.io/projected/080b5369-dcb9-491c-968b-e209b8b6a807-kube-api-access-r8dhv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:53:37.267210 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.267208    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080b5369-dcb9-491c-968b-e209b8b6a807-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:53:37.284398 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.284357    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:37.285268 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.285242    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:37.286837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.286816    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 22:53:37.790085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.790059    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"]
Oct 29 22:53:37.790534 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.790521    4827 cpu_manager.go:414] "containerMap: removing container" podUID="080b5369-dcb9-491c-968b-e209b8b6a807" containerName="revoke-keys"
Oct 29 22:53:37.790577 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.790535    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080b5369-dcb9-491c-968b-e209b8b6a807" containerName="revoke-keys"
Oct 29 22:53:37.790621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.790612    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="080b5369-dcb9-491c-968b-e209b8b6a807" containerName="revoke-keys"
Oct 29 22:53:37.793995 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.793972    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.814257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.814230    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"e2e-conflict-b-2f81a573-kserve-self-signed-certs\""
Oct 29 22:53:37.815564 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.815545    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"]
Oct 29 22:53:37.874489 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.874459    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tmp-dir\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.874648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.874509    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hflct\" (UniqueName: \"kubernetes.io/projected/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-kube-api-access-hflct\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.874704 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.874642    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-home\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.874704 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.874682    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tls-certs\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.874814 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.874736    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-dshm\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.874956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.874941    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-model-cache\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.909720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.909645    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"
Oct 29 22:53:37.909720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.909663    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw" event={"ID":"080b5369-dcb9-491c-968b-e209b8b6a807","Type":"ContainerDied","Data":"dbc12cab949e06fb5b69903435b2cd6573f4691928705b3cd2167201e4bbf95b"}
Oct 29 22:53:37.909958 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.909724    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc12cab949e06fb5b69903435b2cd6573f4691928705b3cd2167201e4bbf95b"
Oct 29 22:53:37.976150 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976121    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-model-cache\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976315 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976200    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tmp-dir\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976315 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976243    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hflct\" (UniqueName: \"kubernetes.io/projected/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-kube-api-access-hflct\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976315 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976294    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-home\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976322    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tls-certs\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976370    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-dshm\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976584 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976560    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-model-cache\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976628    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tmp-dir\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.976718 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.976675    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-home\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.978552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.978528    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-dshm\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:37.978867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:37.978850    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tls-certs\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:38.018789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:38.018606    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hflct\" (UniqueName: \"kubernetes.io/projected/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-kube-api-access-hflct\") pod \"e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:38.110096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:38.110038    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:38.413696 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:38.413670    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"]
Oct 29 22:53:38.415230 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:53:38.415172    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a2d906_b79b_4f6f_a5fc_f7ae33415268.slice/crio-5dc8cc3e7ec690a9c3a5543fb913a54e78c04dd471a5cbad522671c419db4925 WatchSource:0}: Error finding container 5dc8cc3e7ec690a9c3a5543fb913a54e78c04dd471a5cbad522671c419db4925: Status 404 returned error can't find the container with id 5dc8cc3e7ec690a9c3a5543fb913a54e78c04dd471a5cbad522671c419db4925
Oct 29 22:53:38.918595 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:38.918535    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" event={"ID":"b6a2d906-b79b-4f6f-a5fc-f7ae33415268","Type":"ContainerStarted","Data":"97c7396a41a0e93c05cb3397c1678a0e28b57a31f14f97fe0bb5cefafb46aff3"}
Oct 29 22:53:38.918595 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:38.918596    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" event={"ID":"b6a2d906-b79b-4f6f-a5fc-f7ae33415268","Type":"ContainerStarted","Data":"5dc8cc3e7ec690a9c3a5543fb913a54e78c04dd471a5cbad522671c419db4925"}
Oct 29 22:53:38.964547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:38.964500    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" podStartSLOduration=1.9644872370000002 podStartE2EDuration="1.964487237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:38.961642297 +0000 UTC m=+654.584553318" watchObservedRunningTime="2026-10-29 22:53:38.964487237 +0000 UTC m=+654.587398256"
Oct 29 22:53:39.114438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:39.114354    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:48.111003 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:48.110960    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:48.123515 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:48.123491    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:48.985757 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:48.985732    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:53:49.179564 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.179532    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"]
Oct 29 22:53:49.183865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.183836    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.198643 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.198611    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-575d59-openshift-default-dockercfg-gzz6n\""
Oct 29 22:53:49.210048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.209982    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"]
Oct 29 22:53:49.223950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.223921    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.223966    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224133 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.224063    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224234 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.224213    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.224268    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224355 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.224343    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwn9\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-kube-api-access-jdwn9\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224412 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.224381    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224515 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.224501    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.224579 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.224556    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326115 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326033    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326115 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326076    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326115 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326100    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwn9\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-kube-api-access-jdwn9\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326123    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326175    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326221    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326280    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326305    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326338    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326520    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326561    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.326973 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.326950    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.327424 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.327405    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.327511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.327477    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.328885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.328868    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.329456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.329434    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.348840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.348816    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwn9\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-kube-api-access-jdwn9\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.348948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.348842    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.498226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.498195    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:49.813739 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.813715    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"]
Oct 29 22:53:49.814515 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:53:49.814493    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacfb504_ddb6_4ec7_a13d_cc6488c2cc9e.slice/crio-fc6704373ddfcc59f1e41e8ed896105331de9432c932e59160ec29c90ab895c1 WatchSource:0}: Error finding container fc6704373ddfcc59f1e41e8ed896105331de9432c932e59160ec29c90ab895c1: Status 404 returned error can't find the container with id fc6704373ddfcc59f1e41e8ed896105331de9432c932e59160ec29c90ab895c1
Oct 29 22:53:49.816984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.816950    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:49.817071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.817015    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:49.817071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.817065    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:49.984063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.984028    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" event={"ID":"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e","Type":"ContainerStarted","Data":"e6ed3b76546b25e70b6ec67ca0085f3b2cdc1144545451fb87c3b6e7b4a55713"}
Oct 29 22:53:49.984257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:49.984087    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" event={"ID":"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e","Type":"ContainerStarted","Data":"fc6704373ddfcc59f1e41e8ed896105331de9432c932e59160ec29c90ab895c1"}
Oct 29 22:53:50.498842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:50.498813    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:50.500630 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:50.500604    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:53:50.500751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:50.500651    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:53:51.231316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.231235    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podStartSLOduration=2.231221036 podStartE2EDuration="2.231221036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:50.046163925 +0000 UTC m=+665.669074986" watchObservedRunningTime="2026-10-29 22:53:51.231221036 +0000 UTC m=+666.854132062"
Oct 29 22:53:51.232547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.232511    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"]
Oct 29 22:53:51.236541 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.236525    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.260054 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.260022    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"]
Oct 29 22:53:51.327787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327745    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327796    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327829    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327876    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327902    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f7a705-d171-4bd8-a071-4f4ccd575485-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327922    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327949    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qw2\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-kube-api-access-n7qw2\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.327981    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.328034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.328015    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.428958 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.428919    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.428973    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f7a705-d171-4bd8-a071-4f4ccd575485-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.428997    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429022    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qw2\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-kube-api-access-n7qw2\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429074    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429188    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429331    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429442 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429364    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429442 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429396    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429442 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429437    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429681 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429660    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429685    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429761    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.429789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.429785    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f7a705-d171-4bd8-a071-4f4ccd575485-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.431566 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.431546    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.433069 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.433047    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.453045 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.453024    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qw2\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-kube-api-access-n7qw2\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.453149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.453043    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.507624 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.507093    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:53:51.507624 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.507162    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:53:51.551190 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.550943    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:51.923441 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.923416    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"]
Oct 29 22:53:51.924932 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:53:51.924898    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f7a705_d171_4bd8_a071_4f4ccd575485.slice/crio-8d000efd3100fc59c88b58d8692b053fe96252805562fd7d538b40beb2d52bc8 WatchSource:0}: Error finding container 8d000efd3100fc59c88b58d8692b053fe96252805562fd7d538b40beb2d52bc8: Status 404 returned error can't find the container with id 8d000efd3100fc59c88b58d8692b053fe96252805562fd7d538b40beb2d52bc8
Oct 29 22:53:51.928190 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.928156    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:51.928296 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.928221    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:51.928296 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:51.928251    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:53:52.008739 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:52.006860    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" event={"ID":"a0f7a705-d171-4bd8-a071-4f4ccd575485","Type":"ContainerStarted","Data":"8d000efd3100fc59c88b58d8692b053fe96252805562fd7d538b40beb2d52bc8"}
Oct 29 22:53:52.499409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:52.499360    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:53:52.499585 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:52.499422    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:53:53.015732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:53.015696    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" event={"ID":"a0f7a705-d171-4bd8-a071-4f4ccd575485","Type":"ContainerStarted","Data":"2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162"}
Oct 29 22:53:53.098785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:53.095584    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podStartSLOduration=2.095564002 podStartE2EDuration="2.095564002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:53.074260362 +0000 UTC m=+668.697171383" watchObservedRunningTime="2026-10-29 22:53:53.095564002 +0000 UTC m=+668.718475026"
Oct 29 22:53:53.512788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:53.512195    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:53:53.512788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:53.512258    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:53:53.552231 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:53.552191    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:53:53.553457 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:53.553434    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:53:53.553593 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:53.553473    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:53:54.507789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:54.507189    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:53:54.507789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:54.507247    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:53:54.571791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:54.564393    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:53:54.571791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:54.564447    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:53:55.499792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:55.499193    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:53:55.499792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:55.499256    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:53:55.552112 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:55.552073    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:53:55.552488 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:55.552138    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:53:56.504373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:56.500894    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:53:56.504373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:56.500951    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:53:56.552135 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:56.552101    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:53:56.552526 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:56.552157    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:53:57.576844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:57.562184    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:53:57.576844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:57.562242    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:53:58.547795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.546869    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:53:58.547795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.546939    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:58.562790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.559986    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:53:58.562790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.560056    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:53:58.586140 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.584848    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:53:58.590541 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.589832    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:53:58.590740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.590716    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 29 22:53:58.835040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.834913    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:58.848677 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.848646    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:58.850233 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.850213    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:58.852013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.851968    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 22:53:58.934788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.933855    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:58.982088 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.982048    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" podUID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:58.983353 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:58.983319    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:59.142634 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:59.142592    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" podUID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:59.193973 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:59.193899    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:59.264527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:59.264481    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:59.552616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:59.552542    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:53:59.552616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:53:59.552591    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:54:00.556029 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:00.555994    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:54:00.556476 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:00.556460    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:54:00.557225 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:00.557204    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 22:54:02.824733 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:54:02.824699    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:54:14.801924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.801893    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"]
Oct 29 22:54:14.812309 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.812284    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.837425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.835992    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"]
Oct 29 22:54:14.940897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.940862    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.940915    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.940993    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.941044    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.941066    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.941169    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941281 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.941263    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941313 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.941285    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thcq6\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-kube-api-access-thcq6\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:14.941347 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:14.941320    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.041975    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042038    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042061    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042099    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042117    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042188    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042292    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042363    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thcq6\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-kube-api-access-thcq6\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042395    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042418    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042438    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042713 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042630    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.042987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.042959    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.043186 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.043167    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.044411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.044379    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.044653 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.044635    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.068068 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.067996    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.074776 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.074732    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thcq6\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-kube-api-access-thcq6\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.128259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.128228    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:15.446452 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.446423    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"]
Oct 29 22:54:15.447308 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:54:15.447277    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3293ebd3_22cd_4cec_b069_eb54d6687ec9.slice/crio-d4ed78b858d405e2bd77e8c3ace1380d1778ac530f7aba60c52108fc37ce7fa0 WatchSource:0}: Error finding container d4ed78b858d405e2bd77e8c3ace1380d1778ac530f7aba60c52108fc37ce7fa0: Status 404 returned error can't find the container with id d4ed78b858d405e2bd77e8c3ace1380d1778ac530f7aba60c52108fc37ce7fa0
Oct 29 22:54:15.449703 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.449668    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:15.449838 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.449744    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:15.449912 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:15.449845    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:16.356815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.356743    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" event={"ID":"3293ebd3-22cd-4cec-b069-eb54d6687ec9","Type":"ContainerStarted","Data":"cd16e51eab223f756d9dca702be3ca3424a853b8f6e6424b5426af37b119b4ef"}
Oct 29 22:54:16.357285 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.356832    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" event={"ID":"3293ebd3-22cd-4cec-b069-eb54d6687ec9","Type":"ContainerStarted","Data":"d4ed78b858d405e2bd77e8c3ace1380d1778ac530f7aba60c52108fc37ce7fa0"}
Oct 29 22:54:16.394610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.394569    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"]
Oct 29 22:54:16.400131 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.400106    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.410037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.408978    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"]
Oct 29 22:54:16.414921 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.414889    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"distinct-w2-575d59-kserve-self-signed-certs\""
Oct 29 22:54:16.429414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.429360    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podStartSLOduration=2.429346635 podStartE2EDuration="2.429346635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:16.419216441 +0000 UTC m=+692.042127460" watchObservedRunningTime="2026-10-29 22:54:16.429346635 +0000 UTC m=+692.052257667"
Oct 29 22:54:16.461438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.461408    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-dshm\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.461587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.461493    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-home\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.461637 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.461597    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-model-cache\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.461673 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.461645    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-tmp-dir\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.461673 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.461664    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1833b55-765c-43e9-b608-a1c9ca16ca37-tls-certs\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.461756 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.461693    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5nsz\" (UniqueName: \"kubernetes.io/projected/a1833b55-765c-43e9-b608-a1c9ca16ca37-kube-api-access-j5nsz\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.632724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.630856    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-dshm\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.632724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.631048    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-home\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.632724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.631222    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-model-cache\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.632724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.631287    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-tmp-dir\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.632724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.631318    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1833b55-765c-43e9-b608-a1c9ca16ca37-tls-certs\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.632724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.631375    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5nsz\" (UniqueName: \"kubernetes.io/projected/a1833b55-765c-43e9-b608-a1c9ca16ca37-kube-api-access-j5nsz\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.656351 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.656299    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-model-cache\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.656714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.656691    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-tmp-dir\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.724789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.677597    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-home\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.748409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.685980    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5nsz\" (UniqueName: \"kubernetes.io/projected/a1833b55-765c-43e9-b608-a1c9ca16ca37-kube-api-access-j5nsz\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.748679 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.686188    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-dshm\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:16.748810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:16.696702    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1833b55-765c-43e9-b608-a1c9ca16ca37-tls-certs\") pod \"distinct-w2-575d59-kserve-678d58cc44-ztkj2\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:17.015928 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:17.015842    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:17.129263 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:17.129236    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:17.130393 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:17.130372    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:54:17.130516 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:17.130428    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:54:17.344432 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:17.344406    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"]
Oct 29 22:54:17.345431 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:54:17.345406    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1833b55_765c_43e9_b608_a1c9ca16ca37.slice/crio-bc6d93704bb3dea4a9797baeaad5e0f9ac0e1390491715fd72b5ade06c80f2c8 WatchSource:0}: Error finding container bc6d93704bb3dea4a9797baeaad5e0f9ac0e1390491715fd72b5ade06c80f2c8: Status 404 returned error can't find the container with id bc6d93704bb3dea4a9797baeaad5e0f9ac0e1390491715fd72b5ade06c80f2c8
Oct 29 22:54:17.392815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:17.390595    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" event={"ID":"a1833b55-765c-43e9-b608-a1c9ca16ca37","Type":"ContainerStarted","Data":"bc6d93704bb3dea4a9797baeaad5e0f9ac0e1390491715fd72b5ade06c80f2c8"}
Oct 29 22:54:18.128927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:18.128892    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:54:18.129136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:18.128951    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:54:18.397700 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:18.397613    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" event={"ID":"a1833b55-765c-43e9-b608-a1c9ca16ca37","Type":"ContainerStarted","Data":"174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd"}
Oct 29 22:54:18.439134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:18.439087    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podStartSLOduration=2.439074046 podStartE2EDuration="2.439074046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:18.438259838 +0000 UTC m=+694.061170858" watchObservedRunningTime="2026-10-29 22:54:18.439074046 +0000 UTC m=+694.061985065"
Oct 29 22:54:19.016623 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:19.016590    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:19.128708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:19.128676    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:54:19.128910 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:19.128726    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:54:20.129494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:20.129031    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:54:20.129494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:20.129088    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:54:21.139789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:21.132206    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:54:21.139789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:21.132269    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:54:22.128671 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:22.128639    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:54:22.128876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:22.128685    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:54:23.306829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:23.302015    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:23.306829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:23.304539    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:23.306829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:23.306489    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 22:54:27.016260 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:27.016223    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:27.028651 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:27.028624    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:27.496505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:27.496479    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:54:30.427224 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:30.426385    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"]
Oct 29 22:54:32.822987 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:54:32.822951    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:54:35.873464 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:35.873431    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"]
Oct 29 22:54:35.878951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:35.878925    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:35.910640 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:35.910597    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"]
Oct 29 22:54:36.014138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014083    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014213    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e72a9-9c4c-4522-81bc-479aac358efe-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014292    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgjk\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-kube-api-access-ckgjk\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014431    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014499 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014482    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014623 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014594    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e72a9-9c4c-4522-81bc-479aac358efe-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014687 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014671    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014804    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.014897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.014871    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.116356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.116320    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.116550 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.116455    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e72a9-9c4c-4522-81bc-479aac358efe-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.116550 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.116498    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgjk\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-kube-api-access-ckgjk\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.116952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.116828    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.116980    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.117037    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.117078    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e72a9-9c4c-4522-81bc-479aac358efe-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.117139    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117303 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.117209    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117303 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.117280    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.117385    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.117656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.117630    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.118030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.118005    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.118316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.118292    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e72a9-9c4c-4522-81bc-479aac358efe-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.119332 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.119305    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e72a9-9c4c-4522-81bc-479aac358efe-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.120438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.120411    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.144482 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.144427    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.144571 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.144555    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgjk\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-kube-api-access-ckgjk\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.197924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.197891    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:36.525548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.525524    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"]
Oct 29 22:54:36.527263 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:54:36.527237    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a WatchSource:0}: Error finding container 3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a: Status 404 returned error can't find the container with id 3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a
Oct 29 22:54:36.529660 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.529625    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:36.529731 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.529697    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:36.529787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.529727    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:36.549234 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:36.549207    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" event={"ID":"567e72a9-9c4c-4522-81bc-479aac358efe","Type":"ContainerStarted","Data":"3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a"}
Oct 29 22:54:37.557548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:37.557515    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" event={"ID":"567e72a9-9c4c-4522-81bc-479aac358efe","Type":"ContainerStarted","Data":"8269de70e3b79770f9b2c8a2fcf24a8ce478130576fe0103217e86574710a6ed"}
Oct 29 22:54:37.642400 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:37.642335    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podStartSLOduration=2.642316004 podStartE2EDuration="2.642316004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:37.628849036 +0000 UTC m=+713.251760057" watchObservedRunningTime="2026-10-29 22:54:37.642316004 +0000 UTC m=+713.265227025"
Oct 29 22:54:38.198257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:38.198229    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:38.199491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:38.199462    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:54:38.199578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:38.199509    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:54:39.199037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:39.199005    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:54:39.199419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:39.199056    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:54:40.206689 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:40.198929    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:54:40.206689 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:40.198989    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:54:41.004614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.004582    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"]
Oct 29 22:54:41.008818 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.008798    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.025182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.025160    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:41.028998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.028972    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"]
Oct 29 22:54:41.037199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.037176    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:54:41.112544 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.112507    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.112720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.112616    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtltb\" (UniqueName: \"kubernetes.io/projected/3319ba61-f527-4b57-af00-ede4288da8dd-kube-api-access-rtltb\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.112720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.112689    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.199251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.199217    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:54:41.199407 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.199279    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:54:41.213929 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.213900    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.214374 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.213963    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtltb\" (UniqueName: \"kubernetes.io/projected/3319ba61-f527-4b57-af00-ede4288da8dd-kube-api-access-rtltb\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.214374 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.214003    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.214750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.214722    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.216425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.216404    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.237323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.237301    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtltb\" (UniqueName: \"kubernetes.io/projected/3319ba61-f527-4b57-af00-ede4288da8dd-kube-api-access-rtltb\") pod \"maas-api-revoke-sub-36588a39-cfmp5\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.322874 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.322796    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:41.648454 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:41.648429    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"]
Oct 29 22:54:41.649535 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:54:41.649511    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3319ba61_f527_4b57_af00_ede4288da8dd.slice/crio-0b60f09c8ca417394d42d08de4716d45763140ae1e625d194166182e908911e3 WatchSource:0}: Error finding container 0b60f09c8ca417394d42d08de4716d45763140ae1e625d194166182e908911e3: Status 404 returned error can't find the container with id 0b60f09c8ca417394d42d08de4716d45763140ae1e625d194166182e908911e3
Oct 29 22:54:42.199299 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:42.199262    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:54:42.199478 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:42.199319    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:54:42.613411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:42.613378    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5" event={"ID":"3319ba61-f527-4b57-af00-ede4288da8dd","Type":"ContainerStarted","Data":"acb7b7ebe2e75119aa4292ee606b27efd03b9dd108d100b1f8f045940c5f548b"}
Oct 29 22:54:42.613851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:42.613429    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5" event={"ID":"3319ba61-f527-4b57-af00-ede4288da8dd","Type":"ContainerStarted","Data":"0b60f09c8ca417394d42d08de4716d45763140ae1e625d194166182e908911e3"}
Oct 29 22:54:42.655331 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:42.655280    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5" podStartSLOduration=2.65526674 podStartE2EDuration="2.65526674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:42.651935851 +0000 UTC m=+718.274846873" watchObservedRunningTime="2026-10-29 22:54:42.65526674 +0000 UTC m=+718.278177760"
Oct 29 22:54:43.198515 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:43.198483    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:54:43.198693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:43.198531    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:54:43.621193 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:43.621101    4827 generic.go:356] "Generic (PLEG): container finished" podID="3319ba61-f527-4b57-af00-ede4288da8dd" containerID="acb7b7ebe2e75119aa4292ee606b27efd03b9dd108d100b1f8f045940c5f548b" exitCode=0
Oct 29 22:54:43.621576 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:43.621185    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5" event={"ID":"3319ba61-f527-4b57-af00-ede4288da8dd","Type":"ContainerDied","Data":"acb7b7ebe2e75119aa4292ee606b27efd03b9dd108d100b1f8f045940c5f548b"}
Oct 29 22:54:44.203622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.203581    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:44.203899 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.203880    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:44.204669 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.204652    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 22:54:44.782608 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.782584    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:44.971832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.967308    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3319ba61-f527-4b57-af00-ede4288da8dd-kube-api-access-rtltb\" (UniqueName: \"kubernetes.io/projected/3319ba61-f527-4b57-af00-ede4288da8dd-kube-api-access-rtltb\") pod \"3319ba61-f527-4b57-af00-ede4288da8dd\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") "
Oct 29 22:54:44.971832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.967478    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-xks-ca\") pod \"3319ba61-f527-4b57-af00-ede4288da8dd\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") "
Oct 29 22:54:44.971832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.967517    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-openshift-ca\") pod \"3319ba61-f527-4b57-af00-ede4288da8dd\" (UID: \"3319ba61-f527-4b57-af00-ede4288da8dd\") "
Oct 29 22:54:44.971832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.969801    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-openshift-ca" pod "3319ba61-f527-4b57-af00-ede4288da8dd" (UID: "3319ba61-f527-4b57-af00-ede4288da8dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:44.980429 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.980387    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-xks-ca" pod "3319ba61-f527-4b57-af00-ede4288da8dd" (UID: "3319ba61-f527-4b57-af00-ede4288da8dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:44.986479 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:44.986446    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3319ba61-f527-4b57-af00-ede4288da8dd-kube-api-access-rtltb" pod "3319ba61-f527-4b57-af00-ede4288da8dd" (UID: "3319ba61-f527-4b57-af00-ede4288da8dd"). InnerVolumeSpecName "kube-api-access-rtltb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:45.069336 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.069307    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtltb\" (UniqueName: \"kubernetes.io/projected/3319ba61-f527-4b57-af00-ede4288da8dd-kube-api-access-rtltb\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.069336 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.069334    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.069336 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.069344    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3319ba61-f527-4b57-af00-ede4288da8dd-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.463575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.463536    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" podUID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" containerName="main" containerID="cri-o://97c7396a41a0e93c05cb3397c1678a0e28b57a31f14f97fe0bb5cefafb46aff3" gracePeriod=45
Oct 29 22:54:45.639800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.639016    4827 generic.go:356] "Generic (PLEG): container finished" podID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" containerID="97c7396a41a0e93c05cb3397c1678a0e28b57a31f14f97fe0bb5cefafb46aff3" exitCode=0
Oct 29 22:54:45.639800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.639206    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" event={"ID":"b6a2d906-b79b-4f6f-a5fc-f7ae33415268","Type":"ContainerDied","Data":"97c7396a41a0e93c05cb3397c1678a0e28b57a31f14f97fe0bb5cefafb46aff3"}
Oct 29 22:54:45.641804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.641185    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"
Oct 29 22:54:45.641804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.641224    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5" event={"ID":"3319ba61-f527-4b57-af00-ede4288da8dd","Type":"ContainerDied","Data":"0b60f09c8ca417394d42d08de4716d45763140ae1e625d194166182e908911e3"}
Oct 29 22:54:45.641804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.641274    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b60f09c8ca417394d42d08de4716d45763140ae1e625d194166182e908911e3"
Oct 29 22:54:45.705113 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.705088    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:54:45.775498 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775457    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-kube-api-access-hflct\" (UniqueName: \"kubernetes.io/projected/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-kube-api-access-hflct\") pod \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") "
Oct 29 22:54:45.775611 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775528    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-home\") pod \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") "
Oct 29 22:54:45.775611 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775555    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-model-cache\") pod \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") "
Oct 29 22:54:45.775611 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775591    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tmp-dir\") pod \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") "
Oct 29 22:54:45.775785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775633    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tls-certs\") pod \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") "
Oct 29 22:54:45.775785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775759    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-dshm\") pod \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\" (UID: \"b6a2d906-b79b-4f6f-a5fc-f7ae33415268\") "
Oct 29 22:54:45.775895 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775859    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-home" pod "b6a2d906-b79b-4f6f-a5fc-f7ae33415268" (UID: "b6a2d906-b79b-4f6f-a5fc-f7ae33415268"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:45.775950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775892    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-model-cache" pod "b6a2d906-b79b-4f6f-a5fc-f7ae33415268" (UID: "b6a2d906-b79b-4f6f-a5fc-f7ae33415268"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:45.776006 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.775942    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tmp-dir" pod "b6a2d906-b79b-4f6f-a5fc-f7ae33415268" (UID: "b6a2d906-b79b-4f6f-a5fc-f7ae33415268"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:45.776282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.776264    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.776371 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.776285    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.776371 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.776299    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.778462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.778429    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-dshm" pod "b6a2d906-b79b-4f6f-a5fc-f7ae33415268" (UID: "b6a2d906-b79b-4f6f-a5fc-f7ae33415268"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:45.778462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.778449    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tls-certs" pod "b6a2d906-b79b-4f6f-a5fc-f7ae33415268" (UID: "b6a2d906-b79b-4f6f-a5fc-f7ae33415268"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:45.778642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.778500    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-kube-api-access-hflct" pod "b6a2d906-b79b-4f6f-a5fc-f7ae33415268" (UID: "b6a2d906-b79b-4f6f-a5fc-f7ae33415268"). InnerVolumeSpecName "kube-api-access-hflct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:45.877401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.877369    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hflct\" (UniqueName: \"kubernetes.io/projected/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-kube-api-access-hflct\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.877401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.877394    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.877401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:45.877403    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d906-b79b-4f6f-a5fc-f7ae33415268-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:46.264594 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.264563    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"]
Oct 29 22:54:46.265301 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.265276    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3319ba61-f527-4b57-af00-ede4288da8dd" containerName="revoke-keys"
Oct 29 22:54:46.265301 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.265301    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3319ba61-f527-4b57-af00-ede4288da8dd" containerName="revoke-keys"
Oct 29 22:54:46.265472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.265323    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" containerName="main"
Oct 29 22:54:46.265472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.265331    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" containerName="main"
Oct 29 22:54:46.265573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.265472    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3319ba61-f527-4b57-af00-ede4288da8dd" containerName="revoke-keys"
Oct 29 22:54:46.265573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.265487    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" containerName="main"
Oct 29 22:54:46.269343 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.269323    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.281531 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.281505    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tmp-dir\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.281656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.281547    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-dshm\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.281656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.281636    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82cg\" (UniqueName: \"kubernetes.io/projected/77f9fb36-7afe-4473-8a19-b7b8106fff3f-kube-api-access-b82cg\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.281793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.281710    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-home\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.281949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.281926    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-model-cache\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.282096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.282077    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tls-certs\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.284215 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.284161    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"kube-root-ca.crt\""
Oct 29 22:54:46.296125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.296047    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"distinct-2-w0-08e15d-kserve-self-signed-certs\""
Oct 29 22:54:46.296125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.296073    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"openshift-service-ca.crt\""
Oct 29 22:54:46.296125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.296091    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"default-dockercfg-246bt\""
Oct 29 22:54:46.299481 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.299459    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"]
Oct 29 22:54:46.382751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.382720    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-model-cache\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.382991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.382816    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tls-certs\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.382991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.382846    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tmp-dir\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.382991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.382864    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-dshm\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.382991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.382883    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82cg\" (UniqueName: \"kubernetes.io/projected/77f9fb36-7afe-4473-8a19-b7b8106fff3f-kube-api-access-b82cg\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.382991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.382934    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-home\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.383312 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.383215    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-model-cache\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.383312 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.383256    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tmp-dir\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.383312 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.383296    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-home\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.385606 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.385577    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-dshm\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.386493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.386464    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tls-certs\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.406635 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.406608    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82cg\" (UniqueName: \"kubernetes.io/projected/77f9fb36-7afe-4473-8a19-b7b8106fff3f-kube-api-access-b82cg\") pod \"distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.587789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.586335    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:46.655871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.653070    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx" event={"ID":"b6a2d906-b79b-4f6f-a5fc-f7ae33415268","Type":"ContainerDied","Data":"5dc8cc3e7ec690a9c3a5543fb913a54e78c04dd471a5cbad522671c419db4925"}
Oct 29 22:54:46.655871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.653133    4827 scope.go:122] "RemoveContainer" containerID="97c7396a41a0e93c05cb3397c1678a0e28b57a31f14f97fe0bb5cefafb46aff3"
Oct 29 22:54:46.655871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.653325    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"
Oct 29 22:54:46.705572 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.705537    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"]
Oct 29 22:54:46.717108 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.717079    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-b-2f81a573-kserve-6f5ccfcb84-462bx"]
Oct 29 22:54:46.894671 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:46.894647    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"]
Oct 29 22:54:46.895159 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:54:46.895128    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77f9fb36_7afe_4473_8a19_b7b8106fff3f.slice/crio-a0e06de4acc62395eaba8abb1c4684b0792a4d4379c266a97a5817aae6b49adf WatchSource:0}: Error finding container a0e06de4acc62395eaba8abb1c4684b0792a4d4379c266a97a5817aae6b49adf: Status 404 returned error can't find the container with id a0e06de4acc62395eaba8abb1c4684b0792a4d4379c266a97a5817aae6b49adf
Oct 29 22:54:47.662243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:47.662199    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" event={"ID":"77f9fb36-7afe-4473-8a19-b7b8106fff3f","Type":"ContainerStarted","Data":"d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52"}
Oct 29 22:54:47.662441 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:47.662251    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" event={"ID":"77f9fb36-7afe-4473-8a19-b7b8106fff3f","Type":"ContainerStarted","Data":"a0e06de4acc62395eaba8abb1c4684b0792a4d4379c266a97a5817aae6b49adf"}
Oct 29 22:54:47.705123 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:47.705069    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podStartSLOduration=1.705053178 podStartE2EDuration="1.705053178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:47.702705227 +0000 UTC m=+723.325616266" watchObservedRunningTime="2026-10-29 22:54:47.705053178 +0000 UTC m=+723.327964200"
Oct 29 22:54:48.489602 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:48.489571    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a2d906-b79b-4f6f-a5fc-f7ae33415268" path="/var/lib/kubelet/pods/b6a2d906-b79b-4f6f-a5fc-f7ae33415268/volumes"
Oct 29 22:54:48.588199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:48.588173    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:51.422711 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.422680    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"]
Oct 29 22:54:51.427921 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.427899    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.447263 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.447229    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e36796-fc78-472c-89da-3eb09337333f-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.447471 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.447440    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.447571 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.447521    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.447675 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.447656    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.447741 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.447696    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.447859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.447839    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.448158 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.448136    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e36796-fc78-472c-89da-3eb09337333f-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.448355 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.448332    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.448474 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.448379    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjjv\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-kube-api-access-qjjjv\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.464562 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.464537    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"]
Oct 29 22:54:51.549757 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.549726    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e36796-fc78-472c-89da-3eb09337333f-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.549974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.549811    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.549974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.549835    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.549974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.549878    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550156 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550005    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550156 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550105    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550156 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550146    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e36796-fc78-472c-89da-3eb09337333f-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550179    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550218    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjjv\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-kube-api-access-qjjjv\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550264    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550304    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550499 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550329    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550551 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550522    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e36796-fc78-472c-89da-3eb09337333f-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.550551 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.550532    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.552436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.552409    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.552690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.552673    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e36796-fc78-472c-89da-3eb09337333f-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.573240 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.573221    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjjv\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-kube-api-access-qjjjv\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.573347 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.573228    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:51.744673 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:51.744590    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:52.061208 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.061170    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"]
Oct 29 22:54:52.062114 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:54:52.062085    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e36796_fc78_472c_89da_3eb09337333f.slice/crio-ef592e3f8e363c2f32ee57d47dd025a1a4bdb759e8b005990ab626e28e7b535a WatchSource:0}: Error finding container ef592e3f8e363c2f32ee57d47dd025a1a4bdb759e8b005990ab626e28e7b535a: Status 404 returned error can't find the container with id ef592e3f8e363c2f32ee57d47dd025a1a4bdb759e8b005990ab626e28e7b535a
Oct 29 22:54:52.064860 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.064825    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:52.064962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.064899    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:52.064962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.064930    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:54:52.698741 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.698697    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" event={"ID":"74e36796-fc78-472c-89da-3eb09337333f","Type":"ContainerStarted","Data":"85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f"}
Oct 29 22:54:52.699176 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.698782    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" event={"ID":"74e36796-fc78-472c-89da-3eb09337333f","Type":"ContainerStarted","Data":"ef592e3f8e363c2f32ee57d47dd025a1a4bdb759e8b005990ab626e28e7b535a"}
Oct 29 22:54:52.745204 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.745172    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:52.758664 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:52.758616    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podStartSLOduration=1.758601799 podStartE2EDuration="1.758601799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:52.757261502 +0000 UTC m=+728.380172532" watchObservedRunningTime="2026-10-29 22:54:52.758601799 +0000 UTC m=+728.381512819"
Oct 29 22:54:53.746287 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:53.746256    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:54:53.746690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:53.746315    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 29 22:54:54.745216 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:54.745177    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:54:54.745401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:54.745255    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 29 22:54:54.943313 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:54.943286    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"]
Oct 29 22:54:54.947492 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:54.947476    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:54.994056 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:54.994028    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:54:54.994237 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:54.994117    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:54.996830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:54.996742    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"]
Oct 29 22:54:55.023928 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.023899    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjxt\" (UniqueName: \"kubernetes.io/projected/0b62ece1-6022-4ffe-88c5-8062881fa212-kube-api-access-ndjxt\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.024106 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.023965    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.024151 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.024104    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.125883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.125847    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.126057 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.125937    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.126130 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.126113    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjxt\" (UniqueName: \"kubernetes.io/projected/0b62ece1-6022-4ffe-88c5-8062881fa212-kube-api-access-ndjxt\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.126527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.126509    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.128115 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.128099    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.149395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.149372    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjxt\" (UniqueName: \"kubernetes.io/projected/0b62ece1-6022-4ffe-88c5-8062881fa212-kube-api-access-ndjxt\") pod \"maas-api-revoke-sub-eeadf9d4-vmxpb\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.261425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.261336    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:55.658884 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.658703    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"]
Oct 29 22:54:55.661803 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:54:55.661777    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b62ece1_6022_4ffe_88c5_8062881fa212.slice/crio-b8f12c54624886ed6214709de01b6abb6631bcc03439da8bfd3ba4a8fc759814 WatchSource:0}: Error finding container b8f12c54624886ed6214709de01b6abb6631bcc03439da8bfd3ba4a8fc759814: Status 404 returned error can't find the container with id b8f12c54624886ed6214709de01b6abb6631bcc03439da8bfd3ba4a8fc759814
Oct 29 22:54:55.718943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.718906    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb" event={"ID":"0b62ece1-6022-4ffe-88c5-8062881fa212","Type":"ContainerStarted","Data":"b8f12c54624886ed6214709de01b6abb6631bcc03439da8bfd3ba4a8fc759814"}
Oct 29 22:54:55.745018 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.744992    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:54:55.745123 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:55.745038    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 29 22:54:56.590792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:56.590557    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:56.621794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:56.616526    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:56.749300 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:56.749268    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:54:56.754867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:56.754830    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb" event={"ID":"0b62ece1-6022-4ffe-88c5-8062881fa212","Type":"ContainerStarted","Data":"7b720d98aaa995aadbbe75c934f62bc85be4923e3f26a6c65ee18b5c6a456eb3"}
Oct 29 22:54:56.755065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:56.755037    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 29 22:54:56.805282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:56.803099    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:54:56.805984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:56.805942    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb" podStartSLOduration=2.805927415 podStartE2EDuration="2.805927415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:56.805002882 +0000 UTC m=+732.427913903" watchObservedRunningTime="2026-10-29 22:54:56.805927415 +0000 UTC m=+732.428838437"
Oct 29 22:54:57.745129 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:57.745098    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:54:57.745500 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:57.745151    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 29 22:54:57.758047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:57.758020    4827 generic.go:356] "Generic (PLEG): container finished" podID="0b62ece1-6022-4ffe-88c5-8062881fa212" containerID="7b720d98aaa995aadbbe75c934f62bc85be4923e3f26a6c65ee18b5c6a456eb3" exitCode=0
Oct 29 22:54:57.758228 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:57.758108    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb" event={"ID":"0b62ece1-6022-4ffe-88c5-8062881fa212","Type":"ContainerDied","Data":"7b720d98aaa995aadbbe75c934f62bc85be4923e3f26a6c65ee18b5c6a456eb3"}
Oct 29 22:54:58.745497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:58.745456    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:54:58.745924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:58.745522    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 29 22:54:59.023365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.023336    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:59.062752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.062722    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-xks-ca\") pod \"0b62ece1-6022-4ffe-88c5-8062881fa212\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") "
Oct 29 22:54:59.062939 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.062804    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-openshift-ca\") pod \"0b62ece1-6022-4ffe-88c5-8062881fa212\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") "
Oct 29 22:54:59.062939 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.062879    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b62ece1-6022-4ffe-88c5-8062881fa212-kube-api-access-ndjxt\" (UniqueName: \"kubernetes.io/projected/0b62ece1-6022-4ffe-88c5-8062881fa212-kube-api-access-ndjxt\") pod \"0b62ece1-6022-4ffe-88c5-8062881fa212\" (UID: \"0b62ece1-6022-4ffe-88c5-8062881fa212\") "
Oct 29 22:54:59.063434 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.063406    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-openshift-ca" pod "0b62ece1-6022-4ffe-88c5-8062881fa212" (UID: "0b62ece1-6022-4ffe-88c5-8062881fa212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:59.065826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.065749    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-xks-ca" pod "0b62ece1-6022-4ffe-88c5-8062881fa212" (UID: "0b62ece1-6022-4ffe-88c5-8062881fa212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:59.069106 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.069065    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b62ece1-6022-4ffe-88c5-8062881fa212-kube-api-access-ndjxt" pod "0b62ece1-6022-4ffe-88c5-8062881fa212" (UID: "0b62ece1-6022-4ffe-88c5-8062881fa212"). InnerVolumeSpecName "kube-api-access-ndjxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:59.164750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.164718    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.164750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.164745    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b62ece1-6022-4ffe-88c5-8062881fa212-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.164750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.164756    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjxt\" (UniqueName: \"kubernetes.io/projected/0b62ece1-6022-4ffe-88c5-8062881fa212-kube-api-access-ndjxt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.776607 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.776577    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"
Oct 29 22:54:59.776607 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.776592    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb" event={"ID":"0b62ece1-6022-4ffe-88c5-8062881fa212","Type":"ContainerDied","Data":"b8f12c54624886ed6214709de01b6abb6631bcc03439da8bfd3ba4a8fc759814"}
Oct 29 22:54:59.777144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.776640    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f12c54624886ed6214709de01b6abb6631bcc03439da8bfd3ba4a8fc759814"
Oct 29 22:54:59.794928 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.794899    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:59.795356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.795326    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:54:59.796977 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:54:59.796956    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 22:55:02.218904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.218854    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.479755 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.478648    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:55:02.488631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.488580    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.507797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.503349    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.568888 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.559849    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.583356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.583305    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.762952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.762851    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.838988 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:02.837865    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:03.013844 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:55:03.010457    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:55:05.263295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.263264    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"]
Oct 29 22:55:05.263968 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.263947    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0b62ece1-6022-4ffe-88c5-8062881fa212" containerName="revoke-keys"
Oct 29 22:55:05.264036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.263972    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b62ece1-6022-4ffe-88c5-8062881fa212" containerName="revoke-keys"
Oct 29 22:55:05.264174 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.264162    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b62ece1-6022-4ffe-88c5-8062881fa212" containerName="revoke-keys"
Oct 29 22:55:05.267469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.267455    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.286256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.286232    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"]
Oct 29 22:55:05.291472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.291451    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95342afb-e460-4454-86ff-460a10966b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.291596 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.291509    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4pz\" (UniqueName: \"kubernetes.io/projected/95342afb-e460-4454-86ff-460a10966b75-kube-api-access-wf4pz\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.291702 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.291689    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95342afb-e460-4454-86ff-460a10966b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.294909 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.294875    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:05.295035 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.294966    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:05.392493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.392466    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95342afb-e460-4454-86ff-460a10966b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.392628 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.392537    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4pz\" (UniqueName: \"kubernetes.io/projected/95342afb-e460-4454-86ff-460a10966b75-kube-api-access-wf4pz\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.392742 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.392724    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95342afb-e460-4454-86ff-460a10966b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.393375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.393351    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95342afb-e460-4454-86ff-460a10966b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.394876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.394858    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95342afb-e460-4454-86ff-460a10966b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.416053 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.416025    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4pz\" (UniqueName: \"kubernetes.io/projected/95342afb-e460-4454-86ff-460a10966b75-kube-api-access-wf4pz\") pod \"maas-api-revoke-sub-8f0b7366-mzkfc\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.581057 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.580982    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:05.884503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.884477    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"]
Oct 29 22:55:05.885081 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:05.885057    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95342afb_e460_4454_86ff_460a10966b75.slice/crio-56e5639a1ecd85991d49c1520af60ed28b09108bd48fde8b154708c4e75f9601 WatchSource:0}: Error finding container 56e5639a1ecd85991d49c1520af60ed28b09108bd48fde8b154708c4e75f9601: Status 404 returned error can't find the container with id 56e5639a1ecd85991d49c1520af60ed28b09108bd48fde8b154708c4e75f9601
Oct 29 22:55:05.968328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:05.968294    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc" event={"ID":"95342afb-e460-4454-86ff-460a10966b75","Type":"ContainerStarted","Data":"56e5639a1ecd85991d49c1520af60ed28b09108bd48fde8b154708c4e75f9601"}
Oct 29 22:55:06.975830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:06.975729    4827 generic.go:356] "Generic (PLEG): container finished" podID="95342afb-e460-4454-86ff-460a10966b75" containerID="0d11f9fc5287dfdea29e3c00a5c1bfa3fe386412e5968e410400d6ac6adb73b5" exitCode=0
Oct 29 22:55:06.975830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:06.975808    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc" event={"ID":"95342afb-e460-4454-86ff-460a10966b75","Type":"ContainerDied","Data":"0d11f9fc5287dfdea29e3c00a5c1bfa3fe386412e5968e410400d6ac6adb73b5"}
Oct 29 22:55:07.497131 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.497095    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"]
Oct 29 22:55:07.502075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.502050    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.520270 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.520239    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"]
Oct 29 22:55:07.527905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.527879    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"distinct-w1-e131af-kserve-self-signed-certs\""
Oct 29 22:55:07.624673 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.624634    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-home\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.624673 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.624673    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-tmp-dir\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.624948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.624829    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-dshm\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.625001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.624951    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea51e23a-12d9-46d6-9445-5b48cc12459d-tls-certs\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.625035 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.625007    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5jbj\" (UniqueName: \"kubernetes.io/projected/ea51e23a-12d9-46d6-9445-5b48cc12459d-kube-api-access-q5jbj\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.625086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.625040    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-model-cache\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.726051    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-home\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.726103    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-tmp-dir\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.726179    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-dshm\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.726271    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea51e23a-12d9-46d6-9445-5b48cc12459d-tls-certs\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.726346    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5jbj\" (UniqueName: \"kubernetes.io/projected/ea51e23a-12d9-46d6-9445-5b48cc12459d-kube-api-access-q5jbj\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.726396    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-model-cache\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.726916    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-model-cache\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.727180    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-home\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.737828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.727678    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-tmp-dir\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.765833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.764159    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea51e23a-12d9-46d6-9445-5b48cc12459d-tls-certs\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.793554 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.776104    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5jbj\" (UniqueName: \"kubernetes.io/projected/ea51e23a-12d9-46d6-9445-5b48cc12459d-kube-api-access-q5jbj\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.799245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.797088    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-dshm\") pod \"distinct-w1-e131af-kserve-85487c6d9-9vvhm\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:07.830291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:07.830253    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:08.165847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.165826    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"]
Oct 29 22:55:08.167935 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:08.167908    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea51e23a_12d9_46d6_9445_5b48cc12459d.slice/crio-48764452d1abb2cea8dc3687d1e531a7f14ca36de058460232a5ccd84c65eba3 WatchSource:0}: Error finding container 48764452d1abb2cea8dc3687d1e531a7f14ca36de058460232a5ccd84c65eba3: Status 404 returned error can't find the container with id 48764452d1abb2cea8dc3687d1e531a7f14ca36de058460232a5ccd84c65eba3
Oct 29 22:55:08.191250 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.191228    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:08.265335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.265307    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95342afb-e460-4454-86ff-460a10966b75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95342afb-e460-4454-86ff-460a10966b75-maas-api-xks-ca\") pod \"95342afb-e460-4454-86ff-460a10966b75\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") "
Oct 29 22:55:08.265486 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.265469    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95342afb-e460-4454-86ff-460a10966b75-kube-api-access-wf4pz\" (UniqueName: \"kubernetes.io/projected/95342afb-e460-4454-86ff-460a10966b75-kube-api-access-wf4pz\") pod \"95342afb-e460-4454-86ff-460a10966b75\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") "
Oct 29 22:55:08.265538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.265519    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95342afb-e460-4454-86ff-460a10966b75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95342afb-e460-4454-86ff-460a10966b75-maas-api-openshift-ca\") pod \"95342afb-e460-4454-86ff-460a10966b75\" (UID: \"95342afb-e460-4454-86ff-460a10966b75\") "
Oct 29 22:55:08.266335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.266294    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95342afb-e460-4454-86ff-460a10966b75-maas-api-openshift-ca" pod "95342afb-e460-4454-86ff-460a10966b75" (UID: "95342afb-e460-4454-86ff-460a10966b75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:08.267801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.267746    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95342afb-e460-4454-86ff-460a10966b75-maas-api-xks-ca" pod "95342afb-e460-4454-86ff-460a10966b75" (UID: "95342afb-e460-4454-86ff-460a10966b75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:08.267932 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.267752    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95342afb-e460-4454-86ff-460a10966b75-kube-api-access-wf4pz" pod "95342afb-e460-4454-86ff-460a10966b75" (UID: "95342afb-e460-4454-86ff-460a10966b75"). InnerVolumeSpecName "kube-api-access-wf4pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:08.367573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.367497    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95342afb-e460-4454-86ff-460a10966b75-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:08.367573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.367523    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4pz\" (UniqueName: \"kubernetes.io/projected/95342afb-e460-4454-86ff-460a10966b75-kube-api-access-wf4pz\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:08.367573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:08.367534    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95342afb-e460-4454-86ff-460a10966b75-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:09.030610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.030518    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"
Oct 29 22:55:09.030610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.030547    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc" event={"ID":"95342afb-e460-4454-86ff-460a10966b75","Type":"ContainerDied","Data":"56e5639a1ecd85991d49c1520af60ed28b09108bd48fde8b154708c4e75f9601"}
Oct 29 22:55:09.030610 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.030598    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e5639a1ecd85991d49c1520af60ed28b09108bd48fde8b154708c4e75f9601"
Oct 29 22:55:09.033034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.032964    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" event={"ID":"ea51e23a-12d9-46d6-9445-5b48cc12459d","Type":"ContainerStarted","Data":"e98325f42e095b01b51307278226bde121d67d89cb9cfb1142f365cda962c27b"}
Oct 29 22:55:09.033034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.033005    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" event={"ID":"ea51e23a-12d9-46d6-9445-5b48cc12459d","Type":"ContainerStarted","Data":"48764452d1abb2cea8dc3687d1e531a7f14ca36de058460232a5ccd84c65eba3"}
Oct 29 22:55:09.076405 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.076350    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podStartSLOduration=2.076336593 podStartE2EDuration="2.076336593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:09.073821052 +0000 UTC m=+744.696732072" watchObservedRunningTime="2026-10-29 22:55:09.076336593 +0000 UTC m=+744.699247613"
Oct 29 22:55:09.234174 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.233713    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"]
Oct 29 22:55:09.234675 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.234527    4827 cpu_manager.go:414] "containerMap: removing container" podUID="95342afb-e460-4454-86ff-460a10966b75" containerName="revoke-keys"
Oct 29 22:55:09.234675 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.234545    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95342afb-e460-4454-86ff-460a10966b75" containerName="revoke-keys"
Oct 29 22:55:09.234821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.234699    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="95342afb-e460-4454-86ff-460a10966b75" containerName="revoke-keys"
Oct 29 22:55:09.258797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.253038    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.274836 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.269813    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"]
Oct 29 22:55:09.410327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410291    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t7m8\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-kube-api-access-7t7m8\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410482 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410340    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410482 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410411    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410482 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410432    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410539    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410657 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410621    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410694    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410739 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410717    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f0ed7-ce14-4995-be27-a38b35058656-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.410739 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.410734    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512353 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512322    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512353 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512359    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512586 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512431    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512586 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512473    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512586 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512508    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f0ed7-ce14-4995-be27-a38b35058656-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512586 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512533    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512675    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t7m8\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-kube-api-access-7t7m8\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512726    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512942 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512809    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.512942 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.512810    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.513046 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.513026    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.513206 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.513184    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.513291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.513242    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.513347 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.513321    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f0ed7-ce14-4995-be27-a38b35058656-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.514796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.514747    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.514998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.514981    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.535860 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.535837    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.535972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.535901    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t7m8\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-kube-api-access-7t7m8\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.652677 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.638686    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:09.838815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:09.838776    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:10.004339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.004310    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"]
Oct 29 22:55:10.005183 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:10.005155    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5f0ed7_ce14_4995_be27_a38b35058656.slice/crio-0e7af7f70122dc42dfe80c46a95ea8704321938ba912da3ddc8b19e9b70cf1b6 WatchSource:0}: Error finding container 0e7af7f70122dc42dfe80c46a95ea8704321938ba912da3ddc8b19e9b70cf1b6: Status 404 returned error can't find the container with id 0e7af7f70122dc42dfe80c46a95ea8704321938ba912da3ddc8b19e9b70cf1b6
Oct 29 22:55:10.008000 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.007966    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:10.008078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.008058    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:10.008128 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.008112    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:10.044355 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.044323    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" event={"ID":"5a5f0ed7-ce14-4995-be27-a38b35058656","Type":"ContainerStarted","Data":"0e7af7f70122dc42dfe80c46a95ea8704321938ba912da3ddc8b19e9b70cf1b6"}
Oct 29 22:55:10.176451 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.176420    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"]
Oct 29 22:55:10.181285 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.181258    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.204690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.202512    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"]
Oct 29 22:55:10.204690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.203076    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:10.204690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.203076    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:10.330622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.330543    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.330622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.330610    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsj2\" (UniqueName: \"kubernetes.io/projected/cc0b941f-a8d5-48ff-bb94-07afccec1815-kube-api-access-2xsj2\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.331031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.330688    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.431514 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.431476    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsj2\" (UniqueName: \"kubernetes.io/projected/cc0b941f-a8d5-48ff-bb94-07afccec1815-kube-api-access-2xsj2\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.431686 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.431589    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.431867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.431841    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.432383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.432340    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.434189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.434171    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.454585 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.454555    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsj2\" (UniqueName: \"kubernetes.io/projected/cc0b941f-a8d5-48ff-bb94-07afccec1815-kube-api-access-2xsj2\") pod \"maas-api-revoke-sub-18711a78-lthrf\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.496724 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.496695    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:10.823917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:10.823888    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"]
Oct 29 22:55:10.824818 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:10.824791    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0b941f_a8d5_48ff_bb94_07afccec1815.slice/crio-c65d822028f3b3e94c9d5ea8c12276d3cc25f685e0b7a2f0567d189d7c6472d8 WatchSource:0}: Error finding container c65d822028f3b3e94c9d5ea8c12276d3cc25f685e0b7a2f0567d189d7c6472d8: Status 404 returned error can't find the container with id c65d822028f3b3e94c9d5ea8c12276d3cc25f685e0b7a2f0567d189d7c6472d8
Oct 29 22:55:11.052484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:11.052449    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" event={"ID":"5a5f0ed7-ce14-4995-be27-a38b35058656","Type":"ContainerStarted","Data":"816ced82fd1b6237163942c91741aa892253a6e3cc4eb57c3f922ec8f576cc39"}
Oct 29 22:55:11.053821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:11.053796    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf" event={"ID":"cc0b941f-a8d5-48ff-bb94-07afccec1815","Type":"ContainerStarted","Data":"c65d822028f3b3e94c9d5ea8c12276d3cc25f685e0b7a2f0567d189d7c6472d8"}
Oct 29 22:55:11.103231 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:11.103133    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podStartSLOduration=2.103117983 podStartE2EDuration="2.103117983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:11.099908638 +0000 UTC m=+746.722819657" watchObservedRunningTime="2026-10-29 22:55:11.103117983 +0000 UTC m=+746.726029066"
Oct 29 22:55:11.639006 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:11.638976    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:11.640506 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:11.640484    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:11.640584 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:11.640529    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:55:12.062059 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:12.062024    4827 generic.go:356] "Generic (PLEG): container finished" podID="cc0b941f-a8d5-48ff-bb94-07afccec1815" containerID="2ae1907c5cea303c6451fa9963e2379e63d526e72241c8a0499f5da0a5d8a976" exitCode=0
Oct 29 22:55:12.062201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:12.062098    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf" event={"ID":"cc0b941f-a8d5-48ff-bb94-07afccec1815","Type":"ContainerDied","Data":"2ae1907c5cea303c6451fa9963e2379e63d526e72241c8a0499f5da0a5d8a976"}
Oct 29 22:55:12.639632 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:12.639593    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:12.640045 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:12.639653    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:55:13.261569 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.261547    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:13.297037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.297007    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-openshift-ca\") pod \"cc0b941f-a8d5-48ff-bb94-07afccec1815\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") "
Oct 29 22:55:13.297196 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.297051    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-xks-ca\") pod \"cc0b941f-a8d5-48ff-bb94-07afccec1815\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") "
Oct 29 22:55:13.297337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.297318    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0b941f-a8d5-48ff-bb94-07afccec1815-kube-api-access-2xsj2\" (UniqueName: \"kubernetes.io/projected/cc0b941f-a8d5-48ff-bb94-07afccec1815-kube-api-access-2xsj2\") pod \"cc0b941f-a8d5-48ff-bb94-07afccec1815\" (UID: \"cc0b941f-a8d5-48ff-bb94-07afccec1815\") "
Oct 29 22:55:13.297399 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.297347    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-openshift-ca" pod "cc0b941f-a8d5-48ff-bb94-07afccec1815" (UID: "cc0b941f-a8d5-48ff-bb94-07afccec1815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:13.297756 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.297739    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:13.299200 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.299178    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0b941f-a8d5-48ff-bb94-07afccec1815-kube-api-access-2xsj2" pod "cc0b941f-a8d5-48ff-bb94-07afccec1815" (UID: "cc0b941f-a8d5-48ff-bb94-07afccec1815"). InnerVolumeSpecName "kube-api-access-2xsj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:13.299258 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.299218    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-xks-ca" pod "cc0b941f-a8d5-48ff-bb94-07afccec1815" (UID: "cc0b941f-a8d5-48ff-bb94-07afccec1815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:13.398978 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.398944    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsj2\" (UniqueName: \"kubernetes.io/projected/cc0b941f-a8d5-48ff-bb94-07afccec1815-kube-api-access-2xsj2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:13.398978 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.398973    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0b941f-a8d5-48ff-bb94-07afccec1815-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:13.639854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.639822    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:13.640260 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:13.639881    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:55:14.099047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:14.099022    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"
Oct 29 22:55:14.099209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:14.099056    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf" event={"ID":"cc0b941f-a8d5-48ff-bb94-07afccec1815","Type":"ContainerDied","Data":"c65d822028f3b3e94c9d5ea8c12276d3cc25f685e0b7a2f0567d189d7c6472d8"}
Oct 29 22:55:14.099209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:14.099107    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c65d822028f3b3e94c9d5ea8c12276d3cc25f685e0b7a2f0567d189d7c6472d8"
Oct 29 22:55:14.639676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:14.639642    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:14.639891 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:14.639699    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:55:15.639897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:15.639868    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:15.640345 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:15.639916    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:55:16.639984 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:16.639951    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:16.640467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:16.640005    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:55:17.643833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:17.643802    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:17.644313 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:17.644194    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:17.644964 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:17.644945    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 22:55:17.839858 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:17.839826    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:17.853306 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:17.853277    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:18.139009 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:18.138984    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:19.643519 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.643487    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"]
Oct 29 22:55:19.644076 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.644060    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cc0b941f-a8d5-48ff-bb94-07afccec1815" containerName="revoke-keys"
Oct 29 22:55:19.644123 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.644078    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0b941f-a8d5-48ff-bb94-07afccec1815" containerName="revoke-keys"
Oct 29 22:55:19.644206 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.644196    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0b941f-a8d5-48ff-bb94-07afccec1815" containerName="revoke-keys"
Oct 29 22:55:19.648800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.648776    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.666534 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.666508    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"]
Oct 29 22:55:19.674865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.674841    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:19.674975 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.674890    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:19.696907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.696878    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.697126 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.697102    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfp8\" (UniqueName: \"kubernetes.io/projected/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-kube-api-access-lxfp8\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.697341 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.697321    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.799061 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.799026    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.799251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.799078    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.799251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.799204    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfp8\" (UniqueName: \"kubernetes.io/projected/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-kube-api-access-lxfp8\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.799780 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.799745    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.801340 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.801316    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.823566 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.823540    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfp8\" (UniqueName: \"kubernetes.io/projected/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-kube-api-access-lxfp8\") pod \"maas-api-revoke-sub-9bd27b89-8jb2r\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:19.962583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:19.962501    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:20.264126 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:20.264101    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"]
Oct 29 22:55:20.265680 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:20.265652    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb5f503_a4ea_4bc8_9d53_ee3905bdd4bd.slice/crio-b7e6761eee3384b8dba942d49c4e78c81fa36e31483ae6e699952410f017cc35 WatchSource:0}: Error finding container b7e6761eee3384b8dba942d49c4e78c81fa36e31483ae6e699952410f017cc35: Status 404 returned error can't find the container with id b7e6761eee3384b8dba942d49c4e78c81fa36e31483ae6e699952410f017cc35
Oct 29 22:55:21.178800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:21.178754    4827 generic.go:356] "Generic (PLEG): container finished" podID="ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" containerID="05fa6288a902f8272c02d77325b69412d3cc7e005c63ee0a8fda53184b901623" exitCode=0
Oct 29 22:55:21.179153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:21.178835    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r" event={"ID":"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd","Type":"ContainerDied","Data":"05fa6288a902f8272c02d77325b69412d3cc7e005c63ee0a8fda53184b901623"}
Oct 29 22:55:21.179153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:21.178887    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r" event={"ID":"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd","Type":"ContainerStarted","Data":"b7e6761eee3384b8dba942d49c4e78c81fa36e31483ae6e699952410f017cc35"}
Oct 29 22:55:22.385392 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.385299    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:22.432670 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.432633    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-openshift-ca\") pod \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") "
Oct 29 22:55:22.432900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.432720    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-kube-api-access-lxfp8\" (UniqueName: \"kubernetes.io/projected/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-kube-api-access-lxfp8\") pod \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") "
Oct 29 22:55:22.432900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.432808    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-xks-ca\") pod \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\" (UID: \"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd\") "
Oct 29 22:55:22.434827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.434146    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-openshift-ca" pod "ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" (UID: "ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:22.434827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.434522    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"]
Oct 29 22:55:22.435494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.435352    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" containerName="revoke-keys"
Oct 29 22:55:22.435494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.435375    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" containerName="revoke-keys"
Oct 29 22:55:22.435696 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.435598    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" containerName="revoke-keys"
Oct 29 22:55:22.436046 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.435998    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-xks-ca" pod "ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" (UID: "ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:22.437953 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.437913    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-kube-api-access-lxfp8" pod "ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" (UID: "ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd"). InnerVolumeSpecName "kube-api-access-lxfp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:22.450632 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.450479    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.463938 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.463913    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"]
Oct 29 22:55:22.535091 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.535062    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.535235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.535129    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.535235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.535221    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkln\" (UniqueName: \"kubernetes.io/projected/c8754b19-1f30-4e9e-900f-cd099ed18acf-kube-api-access-8tkln\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.535448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.535433    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.535448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.535447    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxfp8\" (UniqueName: \"kubernetes.io/projected/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-kube-api-access-lxfp8\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.535581 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.535456    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.637052 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.637021    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkln\" (UniqueName: \"kubernetes.io/projected/c8754b19-1f30-4e9e-900f-cd099ed18acf-kube-api-access-8tkln\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.637233 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.637173    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.637233 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.637222    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.637807 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.637779    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.639547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.639527    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.659846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.659814    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkln\" (UniqueName: \"kubernetes.io/projected/c8754b19-1f30-4e9e-900f-cd099ed18acf-kube-api-access-8tkln\") pod \"maas-api-revoke-sub-f91a00f4-bvt4q\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:22.773295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:22.773219    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:23.038922 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.038854    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"]
Oct 29 22:55:23.043863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.043843    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.065989 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.065962    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"]
Oct 29 22:55:23.081704 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.081676    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"]
Oct 29 22:55:23.142759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.142733    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.142898 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.142805    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.142898 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.142866    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.143014 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.142937    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.143099 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.143079    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v796z\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-kube-api-access-v796z\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.143157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.143125    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.143191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.143176    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.143227 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.143218    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.143319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.143306    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.192568 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.192545    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"
Oct 29 22:55:23.192706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.192581    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r" event={"ID":"ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd","Type":"ContainerDied","Data":"b7e6761eee3384b8dba942d49c4e78c81fa36e31483ae6e699952410f017cc35"}
Oct 29 22:55:23.192706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.192629    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e6761eee3384b8dba942d49c4e78c81fa36e31483ae6e699952410f017cc35"
Oct 29 22:55:23.194142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.194122    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q" event={"ID":"c8754b19-1f30-4e9e-900f-cd099ed18acf","Type":"ContainerStarted","Data":"9f092e2b812697448d3ed0d604223c0d596ca219ca0458f5a2fb41203298dbd1"}
Oct 29 22:55:23.244634 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.244600    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.244862 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.244660    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.244862 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.244696    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.244862 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.244799    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.244882    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v796z\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-kube-api-access-v796z\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.244920    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.244948    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.245024    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245239 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.245066    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245239 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.245159    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245239 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.245197    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245397 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.245277    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.245397 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.245308    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.246002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.245978    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.247225 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.247201    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.247388 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.247373    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.267616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.267588    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.268083 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.268065    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v796z\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-kube-api-access-v796z\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.361848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.361776    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:23.679354 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.679321    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"]
Oct 29 22:55:23.679828 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:23.679802    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79d5ee2_d16c_4cf4_9fdb_b3b2e55c4c43.slice/crio-e4935693193890dfcdd21ebe66e0c2d57914147d8f28adecfc372b66d4590308 WatchSource:0}: Error finding container e4935693193890dfcdd21ebe66e0c2d57914147d8f28adecfc372b66d4590308: Status 404 returned error can't find the container with id e4935693193890dfcdd21ebe66e0c2d57914147d8f28adecfc372b66d4590308
Oct 29 22:55:23.682389 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.682354    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:23.682491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.682435    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:23.682491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:23.682470    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:24.202863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.202825    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" event={"ID":"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43","Type":"ContainerStarted","Data":"4ee5d9df1dbe5e1e6b27049a361fdf6efe2c96dbd6ab065f6019c9d77a512579"}
Oct 29 22:55:24.203060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.202892    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" event={"ID":"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43","Type":"ContainerStarted","Data":"e4935693193890dfcdd21ebe66e0c2d57914147d8f28adecfc372b66d4590308"}
Oct 29 22:55:24.204633 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.204608    4827 generic.go:356] "Generic (PLEG): container finished" podID="c8754b19-1f30-4e9e-900f-cd099ed18acf" containerID="553e43711ebd456fd5e7666336d4a16f4ddf5b46e9026a2c62ac72535a51fbfe" exitCode=0
Oct 29 22:55:24.204751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.204689    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q" event={"ID":"c8754b19-1f30-4e9e-900f-cd099ed18acf","Type":"ContainerDied","Data":"553e43711ebd456fd5e7666336d4a16f4ddf5b46e9026a2c62ac72535a51fbfe"}
Oct 29 22:55:24.250857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.250742    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podStartSLOduration=1.250723827 podStartE2EDuration="1.250723827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:24.24697557 +0000 UTC m=+759.869886612" watchObservedRunningTime="2026-10-29 22:55:24.250723827 +0000 UTC m=+759.873634848"
Oct 29 22:55:24.362344 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.362314    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:24.363815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.363794    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:24.363915 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:24.363837    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:25.362753 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.362717    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:25.363292 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.362822    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:25.422909 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.422885    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:25.493843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.493674    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8754b19-1f30-4e9e-900f-cd099ed18acf-kube-api-access-8tkln\" (UniqueName: \"kubernetes.io/projected/c8754b19-1f30-4e9e-900f-cd099ed18acf-kube-api-access-8tkln\") pod \"c8754b19-1f30-4e9e-900f-cd099ed18acf\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") "
Oct 29 22:55:25.493843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.493738    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-openshift-ca\") pod \"c8754b19-1f30-4e9e-900f-cd099ed18acf\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") "
Oct 29 22:55:25.495054 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.494990    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-xks-ca\") pod \"c8754b19-1f30-4e9e-900f-cd099ed18acf\" (UID: \"c8754b19-1f30-4e9e-900f-cd099ed18acf\") "
Oct 29 22:55:25.498707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.498463    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-xks-ca" pod "c8754b19-1f30-4e9e-900f-cd099ed18acf" (UID: "c8754b19-1f30-4e9e-900f-cd099ed18acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:25.502404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.502350    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-openshift-ca" pod "c8754b19-1f30-4e9e-900f-cd099ed18acf" (UID: "c8754b19-1f30-4e9e-900f-cd099ed18acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:25.503793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.503745    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8754b19-1f30-4e9e-900f-cd099ed18acf-kube-api-access-8tkln" pod "c8754b19-1f30-4e9e-900f-cd099ed18acf" (UID: "c8754b19-1f30-4e9e-900f-cd099ed18acf"). InnerVolumeSpecName "kube-api-access-8tkln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:25.596703 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.596669    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tkln\" (UniqueName: \"kubernetes.io/projected/c8754b19-1f30-4e9e-900f-cd099ed18acf-kube-api-access-8tkln\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:25.596703 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.596704    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:25.596917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:25.596719    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8754b19-1f30-4e9e-900f-cd099ed18acf-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:26.264792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:26.255618    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"
Oct 29 22:55:26.264792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:26.255830    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q" event={"ID":"c8754b19-1f30-4e9e-900f-cd099ed18acf","Type":"ContainerDied","Data":"9f092e2b812697448d3ed0d604223c0d596ca219ca0458f5a2fb41203298dbd1"}
Oct 29 22:55:26.264792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:26.255882    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f092e2b812697448d3ed0d604223c0d596ca219ca0458f5a2fb41203298dbd1"
Oct 29 22:55:26.366792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:26.365250    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:26.366792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:26.365320    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:27.386848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.365407    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:27.386848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.365481    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:27.598918 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.598871    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:27.599502 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.599471    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:27.625920 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.625876    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:27.769848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.768856    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:27.802148 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.801843    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.006793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:27.978909    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:55:28.034880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.016109    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.134923 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.134871    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:55:28.176907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.176847    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.236951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.236904    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.334850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.329931    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.377792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.367183    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:28.377792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.367244    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:28.478894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.478848    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.497934 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.495851    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.589957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.589854    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.602879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.602835    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.603458 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.603437    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:55:28.632582 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.630848    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.632582 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.631266    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:55:28.869644 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.869548    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:28.953510 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.869885    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:55:28.977866 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.952921    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.058802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.978654    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.058802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:28.980152    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.058802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.056156    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:55:29.058802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.056219    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:55:29.452793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.149388    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.177898    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.282863    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.343862    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.419864    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.487693    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.487740    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.487801    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:55:29.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.487851    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:55:29.596808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.534227    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:29.596808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.534289    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:29.596808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.548854    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.596808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.549189    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:55:29.596808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.550024    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.596808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.550260    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:55:29.619406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.619379    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:55:29.638391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.623182    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.638391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.632866    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.909915 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.909870    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.970714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.970667    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:29.977637 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:29.977607    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:55:30.049883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.049845    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:30.050203 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.050157    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.198006 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.067852    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.199751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.101994    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.430790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.372483    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:30.431139 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.431064    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:30.454842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.384048    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:55:30.482956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.482920    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:55:30.523789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.511186    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:55:30.551905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.551863    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:55:30.552404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.551914    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:30.552601 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.552579    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.588716 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.553733    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.633041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.633000    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.643797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.577131    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:55:30.644287 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.630468    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.865846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.855940    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:30.865846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.856008    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:30.972904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:30.972857    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:55:31.214806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.202890    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:31.277654 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.277623    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:55:31.507902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.503542    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:31.507902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.503643    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:31.512657 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.512628    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:55:31.515588 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.514628    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:31.516282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.516262    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:55:31.534288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.534246    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:55:31.536931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.536905    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:31.547864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.547838    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"]
Oct 29 22:55:31.561508 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.561473    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:55:31.580622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.580235    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c8754b19-1f30-4e9e-900f-cd099ed18acf" containerName="revoke-keys"
Oct 29 22:55:31.580622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.580265    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8754b19-1f30-4e9e-900f-cd099ed18acf" containerName="revoke-keys"
Oct 29 22:55:31.580622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.580567    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8754b19-1f30-4e9e-900f-cd099ed18acf" containerName="revoke-keys"
Oct 29 22:55:31.623047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.619721    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.623047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.622331    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:31.635855 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.635821    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"]
Oct 29 22:55:31.642663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.637976    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:31.642920 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.638540    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:31.643041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.638595    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:31.652930 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.638622    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:31.672591 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.672555    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:55:31.749210 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.749170    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:55:31.750621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.750597    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:55:31.751473 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.751453    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.753593 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.753556    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.753850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.753759    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkg7t\" (UniqueName: \"kubernetes.io/projected/9a6c4480-bc04-476a-85f0-7b716349b07f-kube-api-access-gkg7t\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.804976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.796892    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"]
Oct 29 22:55:31.835082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.835056    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:55:31.836650 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.836633    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:55:31.836915 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.836904    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:55:31.837016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.837006    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"]
Oct 29 22:55:31.837136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.837127    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:55:31.838303 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.838285    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:55:31.838548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.835201    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.842956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.842936    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:55:31.855399 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.855370    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.855519 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.855430    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.855519 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.855497    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkg7t\" (UniqueName: \"kubernetes.io/projected/9a6c4480-bc04-476a-85f0-7b716349b07f-kube-api-access-gkg7t\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.855648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.855549    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8870b-afab-4012-9e54-363671574c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.856049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.856027    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.856143 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.856127    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8870b-afab-4012-9e54-363671574c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.856269 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.856245    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnbf\" (UniqueName: \"kubernetes.io/projected/3ef8870b-afab-4012-9e54-363671574c25-kube-api-access-sjnbf\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.858868 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.858843    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.896574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.896498    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkg7t\" (UniqueName: \"kubernetes.io/projected/9a6c4480-bc04-476a-85f0-7b716349b07f-kube-api-access-gkg7t\") pod \"maas-api-revoke-sub-19baf5dd-t55vz\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:31.958164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.958118    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8870b-afab-4012-9e54-363671574c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.958323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.958215    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnbf\" (UniqueName: \"kubernetes.io/projected/3ef8870b-afab-4012-9e54-363671574c25-kube-api-access-sjnbf\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.958392 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.958379    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8870b-afab-4012-9e54-363671574c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.959753 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.959727    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8870b-afab-4012-9e54-363671574c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:31.961199 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:31.961177    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8870b-afab-4012-9e54-363671574c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:32.004484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.004417    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnbf\" (UniqueName: \"kubernetes.io/projected/3ef8870b-afab-4012-9e54-363671574c25-kube-api-access-sjnbf\") pod \"maas-api-revoke-sub-2ed2912e-khj82\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:32.052415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.052384    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:32.160851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.158398    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:32.362805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.362699    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:32.362805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.362747    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:32.405904 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:32.405870    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6c4480_bc04_476a_85f0_7b716349b07f.slice/crio-22483318ebec6b8aed7cfd94a61caab645344ef2438325dd905912ba14cbf20e WatchSource:0}: Error finding container 22483318ebec6b8aed7cfd94a61caab645344ef2438325dd905912ba14cbf20e: Status 404 returned error can't find the container with id 22483318ebec6b8aed7cfd94a61caab645344ef2438325dd905912ba14cbf20e
Oct 29 22:55:32.405904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.405886    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"]
Oct 29 22:55:32.507856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.507833    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"]
Oct 29 22:55:32.510202 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:32.510175    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef8870b_afab_4012_9e54_363671574c25.slice/crio-3623f3fa81c9e37c859f435b76f008213e6daa673fda8e2fb9abd14a86701570 WatchSource:0}: Error finding container 3623f3fa81c9e37c859f435b76f008213e6daa673fda8e2fb9abd14a86701570: Status 404 returned error can't find the container with id 3623f3fa81c9e37c859f435b76f008213e6daa673fda8e2fb9abd14a86701570
Oct 29 22:55:32.822954 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:55:32.822922    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:55:32.975791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.975171    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz" event={"ID":"9a6c4480-bc04-476a-85f0-7b716349b07f","Type":"ContainerStarted","Data":"99fe289a9a1b6f75e9d089b0a1d141aa0142dfae927e4770c7dc8b4e87f2f16b"}
Oct 29 22:55:32.975791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.975241    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz" event={"ID":"9a6c4480-bc04-476a-85f0-7b716349b07f","Type":"ContainerStarted","Data":"22483318ebec6b8aed7cfd94a61caab645344ef2438325dd905912ba14cbf20e"}
Oct 29 22:55:32.986315 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:32.986235    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82" event={"ID":"3ef8870b-afab-4012-9e54-363671574c25","Type":"ContainerStarted","Data":"3623f3fa81c9e37c859f435b76f008213e6daa673fda8e2fb9abd14a86701570"}
Oct 29 22:55:33.020289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:33.020243    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz" podStartSLOduration=3.020230156 podStartE2EDuration="3.020230156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:33.020037239 +0000 UTC m=+768.642948281" watchObservedRunningTime="2026-10-29 22:55:33.020230156 +0000 UTC m=+768.643141175"
Oct 29 22:55:33.362212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:33.362131    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:33.362212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:33.362194    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:33.996720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:33.996688    4827 generic.go:356] "Generic (PLEG): container finished" podID="9a6c4480-bc04-476a-85f0-7b716349b07f" containerID="99fe289a9a1b6f75e9d089b0a1d141aa0142dfae927e4770c7dc8b4e87f2f16b" exitCode=0
Oct 29 22:55:33.997221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:33.996783    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz" event={"ID":"9a6c4480-bc04-476a-85f0-7b716349b07f","Type":"ContainerDied","Data":"99fe289a9a1b6f75e9d089b0a1d141aa0142dfae927e4770c7dc8b4e87f2f16b"}
Oct 29 22:55:33.998746 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:33.998726    4827 generic.go:356] "Generic (PLEG): container finished" podID="3ef8870b-afab-4012-9e54-363671574c25" containerID="74dfe5fa54920305b9f53a7f6ad480c4478b95ce942d8e1c80430c13c483a207" exitCode=0
Oct 29 22:55:33.998885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:33.998786    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82" event={"ID":"3ef8870b-afab-4012-9e54-363671574c25","Type":"ContainerDied","Data":"74dfe5fa54920305b9f53a7f6ad480c4478b95ce942d8e1c80430c13c483a207"}
Oct 29 22:55:34.362288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:34.362136    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:55:34.362288 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:34.362196    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 22:55:35.166207 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.166183    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:35.211893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.211868    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:35.293563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.293334    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef8870b-afab-4012-9e54-363671574c25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8870b-afab-4012-9e54-363671574c25-maas-api-xks-ca\") pod \"3ef8870b-afab-4012-9e54-363671574c25\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") "
Oct 29 22:55:35.293563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.293428    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef8870b-afab-4012-9e54-363671574c25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8870b-afab-4012-9e54-363671574c25-maas-api-openshift-ca\") pod \"3ef8870b-afab-4012-9e54-363671574c25\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") "
Oct 29 22:55:35.293563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.293493    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6c4480-bc04-476a-85f0-7b716349b07f-kube-api-access-gkg7t\" (UniqueName: \"kubernetes.io/projected/9a6c4480-bc04-476a-85f0-7b716349b07f-kube-api-access-gkg7t\") pod \"9a6c4480-bc04-476a-85f0-7b716349b07f\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") "
Oct 29 22:55:35.293897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.293573    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-openshift-ca\") pod \"9a6c4480-bc04-476a-85f0-7b716349b07f\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") "
Oct 29 22:55:35.293897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.293601    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef8870b-afab-4012-9e54-363671574c25-kube-api-access-sjnbf\" (UniqueName: \"kubernetes.io/projected/3ef8870b-afab-4012-9e54-363671574c25-kube-api-access-sjnbf\") pod \"3ef8870b-afab-4012-9e54-363671574c25\" (UID: \"3ef8870b-afab-4012-9e54-363671574c25\") "
Oct 29 22:55:35.293897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.293815    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-xks-ca\") pod \"9a6c4480-bc04-476a-85f0-7b716349b07f\" (UID: \"9a6c4480-bc04-476a-85f0-7b716349b07f\") "
Oct 29 22:55:35.296549 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.294668    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef8870b-afab-4012-9e54-363671574c25-maas-api-openshift-ca" pod "3ef8870b-afab-4012-9e54-363671574c25" (UID: "3ef8870b-afab-4012-9e54-363671574c25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:35.296549 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.296500    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-openshift-ca" pod "9a6c4480-bc04-476a-85f0-7b716349b07f" (UID: "9a6c4480-bc04-476a-85f0-7b716349b07f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:35.296979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.296926    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6c4480-bc04-476a-85f0-7b716349b07f-kube-api-access-gkg7t" pod "9a6c4480-bc04-476a-85f0-7b716349b07f" (UID: "9a6c4480-bc04-476a-85f0-7b716349b07f"). InnerVolumeSpecName "kube-api-access-gkg7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:35.298817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.297652    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef8870b-afab-4012-9e54-363671574c25-maas-api-xks-ca" pod "3ef8870b-afab-4012-9e54-363671574c25" (UID: "3ef8870b-afab-4012-9e54-363671574c25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:35.299124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.299096    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-xks-ca" pod "9a6c4480-bc04-476a-85f0-7b716349b07f" (UID: "9a6c4480-bc04-476a-85f0-7b716349b07f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:35.299630 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.299612    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef8870b-afab-4012-9e54-363671574c25-kube-api-access-sjnbf" pod "3ef8870b-afab-4012-9e54-363671574c25" (UID: "3ef8870b-afab-4012-9e54-363671574c25"). InnerVolumeSpecName "kube-api-access-sjnbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:35.366181 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.366153    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:35.366641 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.366625    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:35.367428 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.367409    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 22:55:35.395668 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.395638    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8870b-afab-4012-9e54-363671574c25-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:35.395668 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.395663    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkg7t\" (UniqueName: \"kubernetes.io/projected/9a6c4480-bc04-476a-85f0-7b716349b07f-kube-api-access-gkg7t\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:35.395869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.395674    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:35.395869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.395684    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnbf\" (UniqueName: \"kubernetes.io/projected/3ef8870b-afab-4012-9e54-363671574c25-kube-api-access-sjnbf\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:35.395869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.395694    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6c4480-bc04-476a-85f0-7b716349b07f-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:35.395869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:35.395703    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8870b-afab-4012-9e54-363671574c25-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:36.025789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:36.016736    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz" event={"ID":"9a6c4480-bc04-476a-85f0-7b716349b07f","Type":"ContainerDied","Data":"22483318ebec6b8aed7cfd94a61caab645344ef2438325dd905912ba14cbf20e"}
Oct 29 22:55:36.025789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:36.016833    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22483318ebec6b8aed7cfd94a61caab645344ef2438325dd905912ba14cbf20e"
Oct 29 22:55:36.025789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:36.017048    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"
Oct 29 22:55:36.025789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:36.019754    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"
Oct 29 22:55:36.025789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:36.020818    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82" event={"ID":"3ef8870b-afab-4012-9e54-363671574c25","Type":"ContainerDied","Data":"3623f3fa81c9e37c859f435b76f008213e6daa673fda8e2fb9abd14a86701570"}
Oct 29 22:55:36.025789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:36.020884    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3623f3fa81c9e37c859f435b76f008213e6daa673fda8e2fb9abd14a86701570"
Oct 29 22:55:37.147591 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.147566    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"]
Oct 29 22:55:37.148338 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.148311    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9a6c4480-bc04-476a-85f0-7b716349b07f" containerName="revoke-keys"
Oct 29 22:55:37.148338 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.148334    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6c4480-bc04-476a-85f0-7b716349b07f" containerName="revoke-keys"
Oct 29 22:55:37.148493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.148387    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3ef8870b-afab-4012-9e54-363671574c25" containerName="revoke-keys"
Oct 29 22:55:37.148493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.148394    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef8870b-afab-4012-9e54-363671574c25" containerName="revoke-keys"
Oct 29 22:55:37.148560 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.148500    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef8870b-afab-4012-9e54-363671574c25" containerName="revoke-keys"
Oct 29 22:55:37.148560 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.148511    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6c4480-bc04-476a-85f0-7b716349b07f" containerName="revoke-keys"
Oct 29 22:55:37.152527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.152508    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.171493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.171468    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"]
Oct 29 22:55:37.178376 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.178350    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:37.178504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.178405    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:37.324847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.324809    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d76f3e7-bf83-4864-8331-994346725912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.325023 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.324927    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d76f3e7-bf83-4864-8331-994346725912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.325104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.325088    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7996w\" (UniqueName: \"kubernetes.io/projected/7d76f3e7-bf83-4864-8331-994346725912-kube-api-access-7996w\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.426135 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.426052    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7996w\" (UniqueName: \"kubernetes.io/projected/7d76f3e7-bf83-4864-8331-994346725912-kube-api-access-7996w\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.426291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.426160    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d76f3e7-bf83-4864-8331-994346725912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.426291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.426204    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d76f3e7-bf83-4864-8331-994346725912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.426901 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.426876    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d76f3e7-bf83-4864-8331-994346725912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.428569 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.428551    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d76f3e7-bf83-4864-8331-994346725912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.502259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.498972    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7996w\" (UniqueName: \"kubernetes.io/projected/7d76f3e7-bf83-4864-8331-994346725912-kube-api-access-7996w\") pod \"maas-api-revoke-sub-d015a89d-tslcb\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.505795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.503016    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:37.833271 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:37.833241    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"]
Oct 29 22:55:37.834068 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:37.834047    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d76f3e7_bf83_4864_8331_994346725912.slice/crio-7a311bb21afbbbc2c52b87d90a9177c5a06b5b947023aa8e66538ec8daad3094 WatchSource:0}: Error finding container 7a311bb21afbbbc2c52b87d90a9177c5a06b5b947023aa8e66538ec8daad3094: Status 404 returned error can't find the container with id 7a311bb21afbbbc2c52b87d90a9177c5a06b5b947023aa8e66538ec8daad3094
Oct 29 22:55:38.034571 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:38.034537    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb" event={"ID":"7d76f3e7-bf83-4864-8331-994346725912","Type":"ContainerStarted","Data":"7a311bb21afbbbc2c52b87d90a9177c5a06b5b947023aa8e66538ec8daad3094"}
Oct 29 22:55:39.043931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:39.043841    4827 generic.go:356] "Generic (PLEG): container finished" podID="7d76f3e7-bf83-4864-8331-994346725912" containerID="ff229a233e9bba8f66e121d9d1404e0e48a9f4a0b27fd54d9eff237842340c48" exitCode=0
Oct 29 22:55:39.043931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:39.043878    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb" event={"ID":"7d76f3e7-bf83-4864-8331-994346725912","Type":"ContainerDied","Data":"ff229a233e9bba8f66e121d9d1404e0e48a9f4a0b27fd54d9eff237842340c48"}
Oct 29 22:55:40.213302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.213279    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:40.372744 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.372721    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d76f3e7-bf83-4864-8331-994346725912-kube-api-access-7996w\" (UniqueName: \"kubernetes.io/projected/7d76f3e7-bf83-4864-8331-994346725912-kube-api-access-7996w\") pod \"7d76f3e7-bf83-4864-8331-994346725912\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") "
Oct 29 22:55:40.372921 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.372894    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d76f3e7-bf83-4864-8331-994346725912-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d76f3e7-bf83-4864-8331-994346725912-maas-api-xks-ca\") pod \"7d76f3e7-bf83-4864-8331-994346725912\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") "
Oct 29 22:55:40.373100 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.373084    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d76f3e7-bf83-4864-8331-994346725912-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d76f3e7-bf83-4864-8331-994346725912-maas-api-openshift-ca\") pod \"7d76f3e7-bf83-4864-8331-994346725912\" (UID: \"7d76f3e7-bf83-4864-8331-994346725912\") "
Oct 29 22:55:40.373492 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.373461    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d76f3e7-bf83-4864-8331-994346725912-maas-api-openshift-ca" pod "7d76f3e7-bf83-4864-8331-994346725912" (UID: "7d76f3e7-bf83-4864-8331-994346725912"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:40.373605 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.373580    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d76f3e7-bf83-4864-8331-994346725912-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:40.374886 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.374863    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d76f3e7-bf83-4864-8331-994346725912-maas-api-xks-ca" pod "7d76f3e7-bf83-4864-8331-994346725912" (UID: "7d76f3e7-bf83-4864-8331-994346725912"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:40.374953 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.374933    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d76f3e7-bf83-4864-8331-994346725912-kube-api-access-7996w" pod "7d76f3e7-bf83-4864-8331-994346725912" (UID: "7d76f3e7-bf83-4864-8331-994346725912"). InnerVolumeSpecName "kube-api-access-7996w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:40.474399 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.474369    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d76f3e7-bf83-4864-8331-994346725912-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:40.474399 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:40.474397    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7996w\" (UniqueName: \"kubernetes.io/projected/7d76f3e7-bf83-4864-8331-994346725912-kube-api-access-7996w\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:41.058537 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:41.058511    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"
Oct 29 22:55:41.058714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:41.058549    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb" event={"ID":"7d76f3e7-bf83-4864-8331-994346725912","Type":"ContainerDied","Data":"7a311bb21afbbbc2c52b87d90a9177c5a06b5b947023aa8e66538ec8daad3094"}
Oct 29 22:55:41.058714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:41.058594    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a311bb21afbbbc2c52b87d90a9177c5a06b5b947023aa8e66538ec8daad3094"
Oct 29 22:55:43.680749 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.680711    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"]
Oct 29 22:55:43.681714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.681690    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7d76f3e7-bf83-4864-8331-994346725912" containerName="revoke-keys"
Oct 29 22:55:43.681714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.681713    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d76f3e7-bf83-4864-8331-994346725912" containerName="revoke-keys"
Oct 29 22:55:43.681915 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.681883    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d76f3e7-bf83-4864-8331-994346725912" containerName="revoke-keys"
Oct 29 22:55:43.686558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.686536    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.701703 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.701679    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f3092a4d-openshift-default-dockercfg-b9jst\""
Oct 29 22:55:43.718708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.718679    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"]
Oct 29 22:55:43.824590 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824555    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxsgs\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-kube-api-access-dxsgs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.824748 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824599    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-credential-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.824748 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824616    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-certs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.824748 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824694    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-podinfo\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.824905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824758    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-envoy\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.824905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824833    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71fcbb11-1511-426f-9e8e-9052a7b55090-istiod-ca-cert\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.824905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824889    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-data\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.825001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.824923    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.825084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.825070    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-token\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932013    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-podinfo\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932246    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-envoy\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932276    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71fcbb11-1511-426f-9e8e-9052a7b55090-istiod-ca-cert\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932309    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-data\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932355    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932513    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-token\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932591    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxsgs\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-kube-api-access-dxsgs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932633    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-credential-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.932883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.932659    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-certs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.933447 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.933306    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-certs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.935844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.934674    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.935844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.935705    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71fcbb11-1511-426f-9e8e-9052a7b55090-istiod-ca-cert\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.936050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.936029    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-data\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.941786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.936822    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-credential-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.941786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.941006    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-envoy\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.974787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.969683    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-token\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.974787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.971524    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxsgs\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-kube-api-access-dxsgs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:43.992790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:43.990306    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-podinfo\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:44.007785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.004481    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:44.412553 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.412523    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"]
Oct 29 22:55:44.417621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.417604    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.426972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.426724    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"]
Oct 29 22:55:44.427496 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:44.427472    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fcbb11_1511_426f_9e8e_9052a7b55090.slice/crio-ad5fe27d46bc1fd63f7587d14270711cae126f32527d01a9252adb205fc8e15f WatchSource:0}: Error finding container ad5fe27d46bc1fd63f7587d14270711cae126f32527d01a9252adb205fc8e15f: Status 404 returned error can't find the container with id ad5fe27d46bc1fd63f7587d14270711cae126f32527d01a9252adb205fc8e15f
Oct 29 22:55:44.430129 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.430098    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:44.430230 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.430167    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:44.430230 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.430215    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:44.432395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.432377    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:44.434129 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.434056    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:44.436243 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.436224    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"]
Oct 29 22:55:44.444529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.444356    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnlq\" (UniqueName: \"kubernetes.io/projected/9849bcba-1060-4fa9-8888-e6611efed472-kube-api-access-dpnlq\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.444529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.444415    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.444708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.444682    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.545517 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.545487    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.545750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.545605    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnlq\" (UniqueName: \"kubernetes.io/projected/9849bcba-1060-4fa9-8888-e6611efed472-kube-api-access-dpnlq\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.545750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.545665    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.546386 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.546353    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.559931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.559909    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:44.569103 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.569082    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.569732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.569712    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnlq\" (UniqueName: \"kubernetes.io/projected/9849bcba-1060-4fa9-8888-e6611efed472-kube-api-access-dpnlq\") pod \"maas-api-revoke-sub-65c1025d-47gsq\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:44.747478 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.747447    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:44.754710 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:44.754685    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:45.058244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:45.058218    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"]
Oct 29 22:55:45.058873 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:45.058853    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9849bcba_1060_4fa9_8888_e6611efed472.slice/crio-0702cc7a0472dd4356288f0dfbaa33d697353bb7de31f5440d5ba234accb499f WatchSource:0}: Error finding container 0702cc7a0472dd4356288f0dfbaa33d697353bb7de31f5440d5ba234accb499f: Status 404 returned error can't find the container with id 0702cc7a0472dd4356288f0dfbaa33d697353bb7de31f5440d5ba234accb499f
Oct 29 22:55:45.151100 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:45.151067    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" event={"ID":"71fcbb11-1511-426f-9e8e-9052a7b55090","Type":"ContainerStarted","Data":"2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3"}
Oct 29 22:55:45.151259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:45.151126    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" event={"ID":"71fcbb11-1511-426f-9e8e-9052a7b55090","Type":"ContainerStarted","Data":"ad5fe27d46bc1fd63f7587d14270711cae126f32527d01a9252adb205fc8e15f"}
Oct 29 22:55:45.156945 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:45.156922    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq" event={"ID":"9849bcba-1060-4fa9-8888-e6611efed472","Type":"ContainerStarted","Data":"0702cc7a0472dd4356288f0dfbaa33d697353bb7de31f5440d5ba234accb499f"}
Oct 29 22:55:45.243208 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:45.243152    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podStartSLOduration=2.243138163 podStartE2EDuration="2.243138163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:45.240329531 +0000 UTC m=+780.863240550" watchObservedRunningTime="2026-10-29 22:55:45.243138163 +0000 UTC m=+780.866049182"
Oct 29 22:55:46.005231 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:46.005199    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:46.010684 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:46.010663    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:46.165708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:46.165678    4827 generic.go:356] "Generic (PLEG): container finished" podID="9849bcba-1060-4fa9-8888-e6611efed472" containerID="c54c68f8b4a9cc666b89254c5a712c5ec9e8150a52199c461092d86cdc951460" exitCode=0
Oct 29 22:55:46.165866 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:46.165799    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq" event={"ID":"9849bcba-1060-4fa9-8888-e6611efed472","Type":"ContainerDied","Data":"c54c68f8b4a9cc666b89254c5a712c5ec9e8150a52199c461092d86cdc951460"}
Oct 29 22:55:46.166133 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:46.166116    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:46.167136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:46.167118    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:55:47.343835 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.343810    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:47.410186 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.410153    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-openshift-ca\") pod \"9849bcba-1060-4fa9-8888-e6611efed472\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") "
Oct 29 22:55:47.410394 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.410296    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9849bcba-1060-4fa9-8888-e6611efed472-kube-api-access-dpnlq\" (UniqueName: \"kubernetes.io/projected/9849bcba-1060-4fa9-8888-e6611efed472-kube-api-access-dpnlq\") pod \"9849bcba-1060-4fa9-8888-e6611efed472\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") "
Oct 29 22:55:47.410455 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.410421    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-xks-ca\") pod \"9849bcba-1060-4fa9-8888-e6611efed472\" (UID: \"9849bcba-1060-4fa9-8888-e6611efed472\") "
Oct 29 22:55:47.411051 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.411009    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-openshift-ca" pod "9849bcba-1060-4fa9-8888-e6611efed472" (UID: "9849bcba-1060-4fa9-8888-e6611efed472"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:47.416643 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.416612    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9849bcba-1060-4fa9-8888-e6611efed472-kube-api-access-dpnlq" pod "9849bcba-1060-4fa9-8888-e6611efed472" (UID: "9849bcba-1060-4fa9-8888-e6611efed472"). InnerVolumeSpecName "kube-api-access-dpnlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:47.416794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.416661    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-xks-ca" pod "9849bcba-1060-4fa9-8888-e6611efed472" (UID: "9849bcba-1060-4fa9-8888-e6611efed472"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:47.534842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.524073    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:47.534842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.524114    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9849bcba-1060-4fa9-8888-e6611efed472-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:47.534842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:47.524130    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpnlq\" (UniqueName: \"kubernetes.io/projected/9849bcba-1060-4fa9-8888-e6611efed472-kube-api-access-dpnlq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:48.180558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.180525    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"
Oct 29 22:55:48.180558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.180553    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq" event={"ID":"9849bcba-1060-4fa9-8888-e6611efed472","Type":"ContainerDied","Data":"0702cc7a0472dd4356288f0dfbaa33d697353bb7de31f5440d5ba234accb499f"}
Oct 29 22:55:48.180864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.180603    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0702cc7a0472dd4356288f0dfbaa33d697353bb7de31f5440d5ba234accb499f"
Oct 29 22:55:48.493453 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.493368    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"]
Oct 29 22:55:48.494099 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.494078    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9849bcba-1060-4fa9-8888-e6611efed472" containerName="revoke-keys"
Oct 29 22:55:48.494172 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.494101    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9849bcba-1060-4fa9-8888-e6611efed472" containerName="revoke-keys"
Oct 29 22:55:48.494279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.494268    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9849bcba-1060-4fa9-8888-e6611efed472" containerName="revoke-keys"
Oct 29 22:55:48.497512 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.497495    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.532271 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.532245    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:48.532403 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.532370    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:55:48.537120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.537098    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"]
Oct 29 22:55:48.539798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.539384    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgk6b\" (UniqueName: \"kubernetes.io/projected/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-kube-api-access-qgk6b\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.539798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.539761    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.539981 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.539898    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.640974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.640944    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgk6b\" (UniqueName: \"kubernetes.io/projected/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-kube-api-access-qgk6b\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.641168 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.641139    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.641251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.641235    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.641752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.641731    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.643507 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.643477    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.664149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.664123    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgk6b\" (UniqueName: \"kubernetes.io/projected/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-kube-api-access-qgk6b\") pod \"maas-api-revoke-sub-4c2baec1-rpb2m\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:48.811174 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:48.811085    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:49.121412 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.121382    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"]
Oct 29 22:55:49.121905 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:49.121870    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a7ac0f_a806_4bd8_bfe8_0c11f0f5dbfd.slice/crio-ae8506559fb56404f318b8e597d741d5ae50dd8a382249b0ab0ccaf7f47f3b74 WatchSource:0}: Error finding container ae8506559fb56404f318b8e597d741d5ae50dd8a382249b0ab0ccaf7f47f3b74: Status 404 returned error can't find the container with id ae8506559fb56404f318b8e597d741d5ae50dd8a382249b0ab0ccaf7f47f3b74
Oct 29 22:55:49.188427 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.188401    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m" event={"ID":"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd","Type":"ContainerStarted","Data":"ae8506559fb56404f318b8e597d741d5ae50dd8a382249b0ab0ccaf7f47f3b74"}
Oct 29 22:55:49.522596 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.522569    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"]
Oct 29 22:55:49.527282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.527262    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.561121 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.561096    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"]
Oct 29 22:55:49.563395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563363    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563471    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563504    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563587    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563628    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563736 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563702    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99488f4-d71f-4173-a8e7-7def27f6298c-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563877 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563859    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563893    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.563962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.563943    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgl9z\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-kube-api-access-zgl9z\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.666429 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666343    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.666429 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666415    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.666663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666433    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.666663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666487    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.666663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666519    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.666663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666573    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99488f4-d71f-4173-a8e7-7def27f6298c-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666699    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666729    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.666792    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgl9z\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-kube-api-access-zgl9z\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.667015    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.667014    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.667247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.667391    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99488f4-d71f-4173-a8e7-7def27f6298c-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.667949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.667566    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.670150 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.670124    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.670504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.670481    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.690221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.690187    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.690394 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.690323    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgl9z\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-kube-api-access-zgl9z\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:49.854587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:49.854554    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:50.196882 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:50.196849    4827 generic.go:356] "Generic (PLEG): container finished" podID="95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" containerID="42a7525d89b11e618378a425fa4bbd82d675bb06c9afb8e85060843c846921d8" exitCode=0
Oct 29 22:55:50.197072 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:50.196934    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m" event={"ID":"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd","Type":"ContainerDied","Data":"42a7525d89b11e618378a425fa4bbd82d675bb06c9afb8e85060843c846921d8"}
Oct 29 22:55:50.339076 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:50.339049    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"]
Oct 29 22:55:50.339450 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:50.339415    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99488f4_d71f_4173_a8e7_7def27f6298c.slice/crio-7750cb302c1c08a78abfbdcde6236c0bb74e6c0b8e329ebe6e67829e24a7d8b9 WatchSource:0}: Error finding container 7750cb302c1c08a78abfbdcde6236c0bb74e6c0b8e329ebe6e67829e24a7d8b9: Status 404 returned error can't find the container with id 7750cb302c1c08a78abfbdcde6236c0bb74e6c0b8e329ebe6e67829e24a7d8b9
Oct 29 22:55:50.341851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:50.341820    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:50.341951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:50.341893    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:50.341951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:50.341937    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:51.240063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.239549    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" event={"ID":"e99488f4-d71f-4173-a8e7-7def27f6298c","Type":"ContainerStarted","Data":"2cd1ee35af340535ea9763552f3699770021f91562983b9c958cc622c2fe5cfe"}
Oct 29 22:55:51.240063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.239613    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" event={"ID":"e99488f4-d71f-4173-a8e7-7def27f6298c","Type":"ContainerStarted","Data":"7750cb302c1c08a78abfbdcde6236c0bb74e6c0b8e329ebe6e67829e24a7d8b9"}
Oct 29 22:55:51.298790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.297721    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podStartSLOduration=2.297701971 podStartE2EDuration="2.297701971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:51.297266285 +0000 UTC m=+786.920177306" watchObservedRunningTime="2026-10-29 22:55:51.297701971 +0000 UTC m=+786.920612992"
Oct 29 22:55:51.450676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.450653    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:51.523960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.523877    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-xks-ca\") pod \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") "
Oct 29 22:55:51.524134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.524105    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-kube-api-access-qgk6b\" (UniqueName: \"kubernetes.io/projected/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-kube-api-access-qgk6b\") pod \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") "
Oct 29 22:55:51.524212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.524196    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-openshift-ca\") pod \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\" (UID: \"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd\") "
Oct 29 22:55:51.524637 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.524603    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-openshift-ca" pod "95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" (UID: "95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:51.524892 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.524869    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:51.527158 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.527132    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-xks-ca" pod "95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" (UID: "95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:51.527403 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.527381    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-kube-api-access-qgk6b" pod "95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" (UID: "95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd"). InnerVolumeSpecName "kube-api-access-qgk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:51.595736 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.595695    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"]
Oct 29 22:55:51.596351 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.596332    4827 cpu_manager.go:414] "containerMap: removing container" podUID="95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" containerName="revoke-keys"
Oct 29 22:55:51.596459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.596352    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" containerName="revoke-keys"
Oct 29 22:55:51.596552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.596539    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" containerName="revoke-keys"
Oct 29 22:55:51.600091 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.600071    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.631944 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.631907    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632102 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.631958    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632102 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632093    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632137    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632185    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632317 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632219    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632317 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632255    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtg9\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-kube-api-access-vmtg9\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632317 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632297    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632431    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.632517 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632501    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgk6b\" (UniqueName: \"kubernetes.io/projected/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-kube-api-access-qgk6b\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:51.632574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.632519    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:55:51.634408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.634380    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"]
Oct 29 22:55:51.733829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.733755    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734015 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.733878    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734015 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.733899    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734015 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.733982    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734015 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734013    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734060    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734090    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734121    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtg9\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-kube-api-access-vmtg9\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734160    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734445 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734296    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734582 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734560    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734853 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734831    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.734853 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734846    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.735013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.734831    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.736734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.736708    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.736839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.736717    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.756706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.756678    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.756706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.756701    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtg9\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-kube-api-access-vmtg9\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:51.885210 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.877131    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:55:51.887937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.887900    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:55:51.888103 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.887963    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 29 22:55:51.933844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:51.933815    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:52.315541 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.315510    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"
Oct 29 22:55:52.364720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.317611    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m" event={"ID":"95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd","Type":"ContainerDied","Data":"ae8506559fb56404f318b8e597d741d5ae50dd8a382249b0ab0ccaf7f47f3b74"}
Oct 29 22:55:52.364720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.363346    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae8506559fb56404f318b8e597d741d5ae50dd8a382249b0ab0ccaf7f47f3b74"
Oct 29 22:55:52.364720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.363375    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"]
Oct 29 22:55:52.380562 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:52.380517    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a80dafa_e4e6_43b0_979e_e91171c0ea24.slice/crio-fa5f6f31b51555ee5a13c9cec8749bb71ec85f1e406a82deadae3877a3ace21a WatchSource:0}: Error finding container fa5f6f31b51555ee5a13c9cec8749bb71ec85f1e406a82deadae3877a3ace21a: Status 404 returned error can't find the container with id fa5f6f31b51555ee5a13c9cec8749bb71ec85f1e406a82deadae3877a3ace21a
Oct 29 22:55:52.405198 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.404602    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:52.405198 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.404698    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:52.405198 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.404741    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:52.875791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.874133    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:55:52.875791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:52.874190    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 29 22:55:53.029327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.029297    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"]
Oct 29 22:55:53.033590 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.033570    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.085821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.081466    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"]
Oct 29 22:55:53.207851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.207627    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.207851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.207703    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.207851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.207785    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.208158 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.207908    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.208158 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.207983    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnsgq\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-kube-api-access-hnsgq\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.208158 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.208106    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.208158 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.208141    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.208365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.208171    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.208365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.208217    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.313940 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.313886    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314156 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.313986    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314156 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314100    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314172    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnsgq\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-kube-api-access-hnsgq\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314334 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314290    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314334 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314325    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314357    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314403    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314545 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314442    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.314953 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.314924    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.315200 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.315179    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.315783 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.315745    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.316201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.315832    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.316201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.316168    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.319587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.319551    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.319700 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.319681    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.326963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.325916    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" event={"ID":"4a80dafa-e4e6-43b0-979e-e91171c0ea24","Type":"ContainerStarted","Data":"fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb"}
Oct 29 22:55:53.326963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.325973    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" event={"ID":"4a80dafa-e4e6-43b0-979e-e91171c0ea24","Type":"ContainerStarted","Data":"fa5f6f31b51555ee5a13c9cec8749bb71ec85f1e406a82deadae3877a3ace21a"}
Oct 29 22:55:53.338242 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.338150    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnsgq\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-kube-api-access-hnsgq\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.338532 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.338500    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.348467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.348446    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:53.722691 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.722640    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podStartSLOduration=2.722624583 podStartE2EDuration="2.722624583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:53.414393037 +0000 UTC m=+789.037304058" watchObservedRunningTime="2026-10-29 22:55:53.722624583 +0000 UTC m=+789.345535603"
Oct 29 22:55:53.723149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.723126    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"]
Oct 29 22:55:53.723255 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:55:53.723229    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea4e64f_9fbe_4d37_a6c0_ffb3f4f71a40.slice/crio-16a9bf2b022351050c7927dd83429b81d2f2f0fed89ed46eaf736515fecfed63 WatchSource:0}: Error finding container 16a9bf2b022351050c7927dd83429b81d2f2f0fed89ed46eaf736515fecfed63: Status 404 returned error can't find the container with id 16a9bf2b022351050c7927dd83429b81d2f2f0fed89ed46eaf736515fecfed63
Oct 29 22:55:53.725737 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.725707    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:53.725924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.725907    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:53.726028 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.726017    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:53.878790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.873161    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:55:53.878790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.873219    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 29 22:55:53.938796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.935073    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:55:53.966789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.951311    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:55:53.966789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:53.951419    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:55:54.334957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.334922    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" event={"ID":"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40","Type":"ContainerStarted","Data":"9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312"}
Oct 29 22:55:54.335334 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.335003    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" event={"ID":"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40","Type":"ContainerStarted","Data":"16a9bf2b022351050c7927dd83429b81d2f2f0fed89ed46eaf736515fecfed63"}
Oct 29 22:55:54.350073 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.350049    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:55:54.350267 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.350243    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:55:54.350323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.350293    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:55:54.888797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.887856    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:55:54.888797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.887912    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 29 22:55:54.940792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.937700    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:55:54.940792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:54.937760    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:55:55.349000 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:55.348910    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:55:55.349000 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:55.348979    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:55:55.880787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:55.880112    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:55:55.880787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:55.880172    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 29 22:55:55.938802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:55.936721    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:55:55.938802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:55.936802    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:55:56.349173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.349089    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:55:56.349173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.349141    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:55:56.590738 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.590675    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podStartSLOduration=3.59065429 podStartE2EDuration="3.59065429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:54.398282353 +0000 UTC m=+790.021193375" watchObservedRunningTime="2026-10-29 22:55:56.59065429 +0000 UTC m=+792.213565312"
Oct 29 22:55:56.627875 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.603737    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"]
Oct 29 22:55:56.658786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.655821    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.705800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.704296    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e4a049-openshift-default-dockercfg-msz4p\""
Oct 29 22:55:56.728900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.728858    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.729282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.729260    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.737798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.732385    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thnx5\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-kube-api-access-thnx5\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.738103 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.738080    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.746795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.742001    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.866119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.866029    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaabce08-691b-4828-8810-c426708e784e-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.866410 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.866389    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.875788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.866671    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaabce08-691b-4828-8810-c426708e784e-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.876063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.876036    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:56.876815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.876784    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:55:56.895303 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.895205    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 29 22:55:56.977024 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.951850    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:55:56.977024 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:56.951910    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:55:57.018089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.018056    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.052788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.036287    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.053160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.053126    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaabce08-691b-4828-8810-c426708e784e-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.059017 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.058991    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.081799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.081736    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaabce08-691b-4828-8810-c426708e784e-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.082009 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.081824    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.082009 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.081880    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.082009 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.081975    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.082207 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.082076    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thnx5\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-kube-api-access-thnx5\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.082709 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.039842    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.083177 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.083151    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaabce08-691b-4828-8810-c426708e784e-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.083433 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.083412    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.083687 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.083664    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.100147 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.100087    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.112786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.112065    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.321896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.312632    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.348141 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.348048    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"]
Oct 29 22:55:57.375070 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.371786    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:55:57.375070 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.371843    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:55:57.534791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.501520    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaabce08-691b-4828-8810-c426708e784e-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.639848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.605862    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:57.639848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.613534    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thnx5\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-kube-api-access-thnx5\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.757441 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.757407    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:55:57.903845 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.889897    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:55:57.903845 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.889988    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 29 22:55:57.962801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.961704    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:55:57.962801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:57.961809    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:55:58.365301 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.365213    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:55:58.365544 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.365514    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:55:58.574795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.571151    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"]
Oct 29 22:55:58.623885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.623828    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:58.624174 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.624149    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:58.624437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.624401    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:55:58.798037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.797997    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" event={"ID":"aaabce08-691b-4828-8810-c426708e784e","Type":"ContainerStarted","Data":"ab643cd0b812dab0376df64a062e52ae0e9c2785f768dee8b6e526fdb3590394"}
Oct 29 22:55:58.942642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.942610    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:55:58.943037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:58.943005    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:55:59.373791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.355637    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:55:59.373791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.355696    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:55:59.556859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.545520    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:59.610787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.586442    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:59.619806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.612944    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:55:59.623943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.623892    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:59.651039 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.636385    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:55:59.753182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.753131    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:59.824530 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.824418    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:55:59.872791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.869856    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:59.969445 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.968549    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded"
Oct 29 22:55:59.969856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.969752    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:55:59.969856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.969810    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:56:00.008793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:55:59.985405    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:00.045924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.034445    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:00.045924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.034514    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:00.102081 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.101979    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:00.129893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.108232    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" event={"ID":"aaabce08-691b-4828-8810-c426708e784e","Type":"ContainerStarted","Data":"30e262dfc9ac5a7e450533f7af75562522ff03aa7d7d22892ddcf7705bfa3713"}
Oct 29 22:56:00.235154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.235095    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podStartSLOduration=4.235075201 podStartE2EDuration="4.235075201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:00.186634621 +0000 UTC m=+795.809545655" watchObservedRunningTime="2026-10-29 22:56:00.235075201 +0000 UTC m=+795.857986222"
Oct 29 22:56:00.354081 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.353987    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.360922 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.360859    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:56:00.573788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.552807    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:00.686665 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.615432    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:00.688950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.615472    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:00.703800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.689453    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:56:00.703800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.689523    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:56:00.703800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.689563    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:56:00.703800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.669866    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:00.703800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.669915    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:00.703800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.669938    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:00.704298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.646637    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:00.708630 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.706860    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:56:00.708630 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.706980    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:56:00.767757 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.767725    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:56:00.793046 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.793000    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.793328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.793284    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:56:00.793559 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.793539    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:00.794032 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.794013    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:56:00.839920 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.839872    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:00.840511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.840490    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:00.889807 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.889156    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:00.890271 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.890243    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:56:00.958598 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.935490    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.977433 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:00.958856    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:56:01.033816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.026884    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:01.033816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.028725    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:01.215573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.215480    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:01.215897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.215864    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:01.418816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.400391    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:56:01.418816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.400443    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:56:01.718813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.690363    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:01.718813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.691192    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:01.718813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.692152    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:01.718813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.712481    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:01.761794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.735542    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:01.761794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.736058    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:56:01.761794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.737802    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:01.761794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.738356    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:01.800990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.800952    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:56:02.005850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.997334    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:56:02.005850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.851184    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:02.005850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:01.834477    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:02.089934 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.089888    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:02.120370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.103905    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:02.361848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.361090    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:02.361848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.361324    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:02.374791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.363861    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.374791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.363914    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:56:02.414176 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.413084    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.414176 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.413159    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:56:02.586792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.580323    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:02.714790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.700673    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:02.715412 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.705845    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:02.715871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.705920    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:02.732790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.719515    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:02.753797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.743326    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:02.753797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.743743    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:02.753797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.744079    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:02.941799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.941019    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.941799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:02.941081    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:56:03.118846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.116183    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.184144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.184094    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.210793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.184879    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:03.286792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.233854    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.312094 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.237349    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.397844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.355223    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:56:03.398057 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.397903    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:56:03.398057 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.358510    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:56:03.425792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.417852    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:03.425792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.417910    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.743857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.718870    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.782287 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.719972    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:03.810791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.720051    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:03.896337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.744899    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.923801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.748236    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:03.924359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.748364    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:03.936787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.748387    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:04.237297 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:56:03.814242    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:56:04.237563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:03.852303    4827 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:04.237738 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.237706    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:04.237906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.151864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:04.242097 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.029019    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:04.245861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.191038    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:04.276882 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.276837    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:04.277134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.277102    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:04.314804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.297409    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:04.376927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.363376    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:56:04.376927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.363429    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:56:04.386992 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.386938    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:04.399957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.399927    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:56:04.493840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.493292    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:04.493840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.493355    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:04.784960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.784856    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:04.818798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.803838    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:04.874792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.870905    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:04.910381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.891014    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:56:04.953922 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.953880    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:56:04.955249 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.955218    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:05.015104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:04.956941    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:05.037621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.037534    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:05.045135 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.045096    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.318175 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.318063    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.347268 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.347228    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:05.454790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.448847    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:05.459708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.459669    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.524790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.521092    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.603850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.592915    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:05.603850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.593004    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.603850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.596812    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:56:05.603850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.596870    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:56:05.869846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.804093    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.869846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.823226    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:05.894788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.894482    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.991787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.960305    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.991787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.960831    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:05.991787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:05.973378    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.101798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.101224    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:06.101798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.101302    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.280167 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.280129    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:06.314595 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.314460    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.316372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.312169    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:56:06.390871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.322679    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.431671 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.416620    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:56:06.431936 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.431902    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 22:56:06.457802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.433132    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.462491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.462464    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:56:06.463495 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.463479    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:56:06.464159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.464126    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.507367 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.506912    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:06.552026 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.548080    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.615120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.615084    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:06.615402 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.615359    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.622144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.620842    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:06.622144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.620881    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.643563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.643535    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 22:56:06.653680 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.649875    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:06.653680 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.649929    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:06.906804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.905844    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:06.906804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.906712    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:07.020040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:06.930094    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:07.109790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.108855    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:07.123993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.123937    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:07.183439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.182164    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:56:07.194101 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.194073    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 22:56:07.346712 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.345632    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:07.446596 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.445874    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:07.528221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.528049    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:07.601340 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.601301    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:07.619929 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.619881    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:07.666649 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.666590    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:07.860689 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.860550    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:56:07.912787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.906636    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:07.912787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:07.906963    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:08.106148 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.106043    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:56:08.106420 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.106400    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:56:08.131266 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.131166    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:08.133883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.133838    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:08.153788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.143157    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:08.153788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.153090    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:08.154350 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.154328    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:56:08.155491 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.155232    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:56:08.240658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.240630    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"]
Oct 29 22:56:08.249609 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.249585    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"]
Oct 29 22:56:08.249878 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.249863    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:56:08.251032 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.251001    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.256465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.256441    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:56:08.267785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.263993    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:56:08.296962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.296464    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"]
Oct 29 22:56:08.301782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.301725    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"]
Oct 29 22:56:08.301929 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.301864    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:08.321086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.321063    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e4a049-dockercfg-tvj2f\""
Oct 29 22:56:08.321226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.321172    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e4a049\""
Oct 29 22:56:08.347335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.346329    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxrn\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-kube-api-access-twxrn\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.347335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.346700    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.347335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.346740    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.347335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.346800    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.348312 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.348284    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.348923 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.348906    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:56:08.349367 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.349335    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.349455 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.349391    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.349455 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.349446    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.349546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.349505    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.355064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.355038    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:08.355176 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.355085    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:08.420804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.420712    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:56:08.451185 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451147    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtclt\" (UniqueName: \"kubernetes.io/projected/62509af9-d4c8-4148-aba9-f996a9dff494-kube-api-access-vtclt\") pod \"payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:08.451385 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451208    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451385 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451333    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxrn\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-kube-api-access-twxrn\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451498 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451465    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62509af9-d4c8-4148-aba9-f996a9dff494-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:08.451648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451630    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451721 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451668    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451721 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451699    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451836 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451789    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451890 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451862    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451901    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.451950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.451942    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.452843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.452817    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.452957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.452877    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.453239 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.453191    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.453490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.453470    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.453906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.453875    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.455620 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.455584    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.456707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.456681    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.475284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.475254    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxrn\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-kube-api-access-twxrn\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.476169 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.476141    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.552980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.552943    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtclt\" (UniqueName: \"kubernetes.io/projected/62509af9-d4c8-4148-aba9-f996a9dff494-kube-api-access-vtclt\") pod \"payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:08.553142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.553057    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62509af9-d4c8-4148-aba9-f996a9dff494-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:08.553725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.553705    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62509af9-d4c8-4148-aba9-f996a9dff494-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:08.575941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.575912    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtclt\" (UniqueName: \"kubernetes.io/projected/62509af9-d4c8-4148-aba9-f996a9dff494-kube-api-access-vtclt\") pod \"payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:08.598498 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.598467    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:08.625044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:08.625017    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:09.139238 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.139207    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"]
Oct 29 22:56:09.160197 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.160149    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:09.160885 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.160853    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:09.161102 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.161070    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:09.280861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.280221    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" event={"ID":"7bf88017-90b5-4dd2-ab04-b1b6b9915066","Type":"ContainerStarted","Data":"9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58"}
Oct 29 22:56:09.280861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.280283    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" event={"ID":"7bf88017-90b5-4dd2-ab04-b1b6b9915066","Type":"ContainerStarted","Data":"14c26a32204a72df0815567fa2bffd70d9a0fe06b94904024106e3e36489a167"}
Oct 29 22:56:09.281540 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.281507    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 22:56:09.309877 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:09.309845    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62509af9_d4c8_4148_aba9_f996a9dff494.slice/crio-a45c6b6bea1082b8645c2e1044bfa99e6ea047d91efc5414ed9e74aad4bd6d59 WatchSource:0}: Error finding container a45c6b6bea1082b8645c2e1044bfa99e6ea047d91efc5414ed9e74aad4bd6d59: Status 404 returned error can't find the container with id a45c6b6bea1082b8645c2e1044bfa99e6ea047d91efc5414ed9e74aad4bd6d59
Oct 29 22:56:09.310061 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.310033    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"]
Oct 29 22:56:09.325381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.325332    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podStartSLOduration=1.325317219 podStartE2EDuration="1.325317219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:09.324369895 +0000 UTC m=+804.947280928" watchObservedRunningTime="2026-10-29 22:56:09.325317219 +0000 UTC m=+804.948228239"
Oct 29 22:56:09.599007 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.598914    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:09.683363 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.683324    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"]
Oct 29 22:56:09.688558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.688534    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.703648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.703623    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:56:09.703803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.703709    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:09.706512 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.706490    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"]
Oct 29 22:56:09.827796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.815074    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.827796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.815167    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kn4\" (UniqueName: \"kubernetes.io/projected/570f1185-c7dd-4098-b05d-ce776a83926e-kube-api-access-q2kn4\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.829514 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.828973    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.931692 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.930711    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.931692 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.930820    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kn4\" (UniqueName: \"kubernetes.io/projected/570f1185-c7dd-4098-b05d-ce776a83926e-kube-api-access-q2kn4\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.931692 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.930951    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.932645 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.932539    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.956527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.956493    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:09.959609 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:09.959446    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kn4\" (UniqueName: \"kubernetes.io/projected/570f1185-c7dd-4098-b05d-ce776a83926e-kube-api-access-q2kn4\") pod \"maas-api-revoke-sub-9f0a3f00-cdcrw\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:10.005290 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.005258    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:10.298458 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.298360    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj" event={"ID":"62509af9-d4c8-4148-aba9-f996a9dff494","Type":"ContainerStarted","Data":"17d92f24c993c140e352f9a6052230c5851bb5f529a3e915ed8da73348726008"}
Oct 29 22:56:10.298458 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.298449    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj" event={"ID":"62509af9-d4c8-4148-aba9-f996a9dff494","Type":"ContainerStarted","Data":"a45c6b6bea1082b8645c2e1044bfa99e6ea047d91efc5414ed9e74aad4bd6d59"}
Oct 29 22:56:10.298965 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.298927    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:10.316026 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:10.316005    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570f1185_c7dd_4098_b05d_ce776a83926e.slice/crio-f82f793f08c5b176c0ab717943d427dce328c112507f017f33c575e0de2da4b4 WatchSource:0}: Error finding container f82f793f08c5b176c0ab717943d427dce328c112507f017f33c575e0de2da4b4: Status 404 returned error can't find the container with id f82f793f08c5b176c0ab717943d427dce328c112507f017f33c575e0de2da4b4
Oct 29 22:56:10.316552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.316530    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"]
Oct 29 22:56:10.350325 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.350282    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj" podStartSLOduration=2.350267194 podStartE2EDuration="2.350267194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:10.3483985 +0000 UTC m=+805.971309520" watchObservedRunningTime="2026-10-29 22:56:10.350267194 +0000 UTC m=+805.973178213"
Oct 29 22:56:10.610841 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.610222    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:56:10.610841 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:10.610311    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:56:11.309012 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:11.308977    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw" event={"ID":"570f1185-c7dd-4098-b05d-ce776a83926e","Type":"ContainerStarted","Data":"4a7d5f67c4fe71a7a118e090a6c0c5e0c5c7196ac5e58f57ae6c7e9e7ae91d62"}
Oct 29 22:56:11.309579 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:11.309030    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw" event={"ID":"570f1185-c7dd-4098-b05d-ce776a83926e","Type":"ContainerStarted","Data":"f82f793f08c5b176c0ab717943d427dce328c112507f017f33c575e0de2da4b4"}
Oct 29 22:56:11.353528 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:11.353480    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw" podStartSLOduration=2.353466581 podStartE2EDuration="2.353466581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:11.348810468 +0000 UTC m=+806.971721484" watchObservedRunningTime="2026-10-29 22:56:11.353466581 +0000 UTC m=+806.976377601"
Oct 29 22:56:11.600852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:11.600274    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:56:11.600852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:11.600349    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:56:12.318963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.318926    4827 generic.go:356] "Generic (PLEG): container finished" podID="570f1185-c7dd-4098-b05d-ce776a83926e" containerID="4a7d5f67c4fe71a7a118e090a6c0c5e0c5c7196ac5e58f57ae6c7e9e7ae91d62" exitCode=0
Oct 29 22:56:12.319486 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.319004    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw" event={"ID":"570f1185-c7dd-4098-b05d-ce776a83926e","Type":"ContainerDied","Data":"4a7d5f67c4fe71a7a118e090a6c0c5e0c5c7196ac5e58f57ae6c7e9e7ae91d62"}
Oct 29 22:56:12.328622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.328595    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"]
Oct 29 22:56:12.332748 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.332733    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.352089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.352063    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"]
Oct 29 22:56:12.442801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.442744    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.443076 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.443054    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4brn\" (UniqueName: \"kubernetes.io/projected/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-kube-api-access-t4brn\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.443160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.443111    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.544461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.544422    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4brn\" (UniqueName: \"kubernetes.io/projected/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-kube-api-access-t4brn\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.544461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.544467    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.544683 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.544597    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.545150 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.545127    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.546837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.546820    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.590641 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.590553    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4brn\" (UniqueName: \"kubernetes.io/projected/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-kube-api-access-t4brn\") pod \"maas-api-revoke-sub-328a78a3-jwr5b\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:12.599651 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.599621    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:56:12.599813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.599682    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:56:12.647472 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:12.647440    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:13.020791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.020163    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"]
Oct 29 22:56:13.091792 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:13.090261    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c13d60d_4d6a_4ce0_ba8b_76eea17002e2.slice/crio-bd37cfb3113b59e084a89fef39aace40f920f483f02d4b3d2ad618c1f6b2e209 WatchSource:0}: Error finding container bd37cfb3113b59e084a89fef39aace40f920f483f02d4b3d2ad618c1f6b2e209: Status 404 returned error can't find the container with id bd37cfb3113b59e084a89fef39aace40f920f483f02d4b3d2ad618c1f6b2e209
Oct 29 22:56:13.329866 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.329752    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b" event={"ID":"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2","Type":"ContainerStarted","Data":"bd37cfb3113b59e084a89fef39aace40f920f483f02d4b3d2ad618c1f6b2e209"}
Oct 29 22:56:13.544553 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.544527    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:13.599580 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.599501    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:56:13.599580 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.599553    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:56:13.686797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.681495    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570f1185-c7dd-4098-b05d-ce776a83926e-kube-api-access-q2kn4\" (UniqueName: \"kubernetes.io/projected/570f1185-c7dd-4098-b05d-ce776a83926e-kube-api-access-q2kn4\") pod \"570f1185-c7dd-4098-b05d-ce776a83926e\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") "
Oct 29 22:56:13.686797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.681618    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-xks-ca\") pod \"570f1185-c7dd-4098-b05d-ce776a83926e\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") "
Oct 29 22:56:13.686797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.681665    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-openshift-ca\") pod \"570f1185-c7dd-4098-b05d-ce776a83926e\" (UID: \"570f1185-c7dd-4098-b05d-ce776a83926e\") "
Oct 29 22:56:13.687999 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.687966    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-openshift-ca" pod "570f1185-c7dd-4098-b05d-ce776a83926e" (UID: "570f1185-c7dd-4098-b05d-ce776a83926e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:13.688533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.688509    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:13.691180 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.691148    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-xks-ca" pod "570f1185-c7dd-4098-b05d-ce776a83926e" (UID: "570f1185-c7dd-4098-b05d-ce776a83926e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:13.691655 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.691628    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570f1185-c7dd-4098-b05d-ce776a83926e-kube-api-access-q2kn4" pod "570f1185-c7dd-4098-b05d-ce776a83926e" (UID: "570f1185-c7dd-4098-b05d-ce776a83926e"). InnerVolumeSpecName "kube-api-access-q2kn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:13.790001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.789974    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2kn4\" (UniqueName: \"kubernetes.io/projected/570f1185-c7dd-4098-b05d-ce776a83926e-kube-api-access-q2kn4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:13.790001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:13.789999    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f1185-c7dd-4098-b05d-ce776a83926e-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:14.338546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:14.338510    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b" event={"ID":"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2","Type":"ContainerStarted","Data":"7e77b4b138d7fd3466adecd7d4b78c5c4a87858dcf83f8ae255e057354ab732d"}
Oct 29 22:56:14.339933 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:14.339913    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"
Oct 29 22:56:14.340073 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:14.339929    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw" event={"ID":"570f1185-c7dd-4098-b05d-ce776a83926e","Type":"ContainerDied","Data":"f82f793f08c5b176c0ab717943d427dce328c112507f017f33c575e0de2da4b4"}
Oct 29 22:56:14.340073 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:14.339967    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f82f793f08c5b176c0ab717943d427dce328c112507f017f33c575e0de2da4b4"
Oct 29 22:56:14.396135 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:14.395993    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b" podStartSLOduration=2.395973424 podStartE2EDuration="2.395973424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:14.39316176 +0000 UTC m=+810.016072780" watchObservedRunningTime="2026-10-29 22:56:14.395973424 +0000 UTC m=+810.018884454"
Oct 29 22:56:14.622842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:14.614197    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:56:14.622842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:14.614258    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:56:15.348960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:15.348928    4827 generic.go:356] "Generic (PLEG): container finished" podID="7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" containerID="7e77b4b138d7fd3466adecd7d4b78c5c4a87858dcf83f8ae255e057354ab732d" exitCode=0
Oct 29 22:56:15.349380 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:15.349009    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b" event={"ID":"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2","Type":"ContainerDied","Data":"7e77b4b138d7fd3466adecd7d4b78c5c4a87858dcf83f8ae255e057354ab732d"}
Oct 29 22:56:15.599231 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:15.599148    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:56:15.599231 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:15.599204    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:56:16.541116 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.541089    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:16.660053 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.660004    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-kube-api-access-t4brn\" (UniqueName: \"kubernetes.io/projected/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-kube-api-access-t4brn\") pod \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") "
Oct 29 22:56:16.660343 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.660313    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-openshift-ca\") pod \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") "
Oct 29 22:56:16.660426 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.660374    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-xks-ca\") pod \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\" (UID: \"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2\") "
Oct 29 22:56:16.660650 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.660626    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-openshift-ca" pod "7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" (UID: "7c13d60d-4d6a-4ce0-ba8b-76eea17002e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:16.661041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.661018    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:16.662212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.662187    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-kube-api-access-t4brn" pod "7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" (UID: "7c13d60d-4d6a-4ce0-ba8b-76eea17002e2"). InnerVolumeSpecName "kube-api-access-t4brn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:16.662425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.662411    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-xks-ca" pod "7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" (UID: "7c13d60d-4d6a-4ce0-ba8b-76eea17002e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:16.762631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.762592    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4brn\" (UniqueName: \"kubernetes.io/projected/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-kube-api-access-t4brn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:16.762631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.762621    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:16.902887 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.902851    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:16.903907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.903515    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:16.905051 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:16.905032    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 22:56:17.368914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:17.368883    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"
Oct 29 22:56:17.369071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:17.368908    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b" event={"ID":"7c13d60d-4d6a-4ce0-ba8b-76eea17002e2","Type":"ContainerDied","Data":"bd37cfb3113b59e084a89fef39aace40f920f483f02d4b3d2ad618c1f6b2e209"}
Oct 29 22:56:17.369071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:17.368963    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd37cfb3113b59e084a89fef39aace40f920f483f02d4b3d2ad618c1f6b2e209"
Oct 29 22:56:18.750614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.750576    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"]
Oct 29 22:56:18.751547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.751525    4827 cpu_manager.go:414] "containerMap: removing container" podUID="570f1185-c7dd-4098-b05d-ce776a83926e" containerName="revoke-keys"
Oct 29 22:56:18.751633 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.751548    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570f1185-c7dd-4098-b05d-ce776a83926e" containerName="revoke-keys"
Oct 29 22:56:18.751633 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.751598    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" containerName="revoke-keys"
Oct 29 22:56:18.751633 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.751605    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" containerName="revoke-keys"
Oct 29 22:56:18.751826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.751794    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" containerName="revoke-keys"
Oct 29 22:56:18.751826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.751817    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="570f1185-c7dd-4098-b05d-ce776a83926e" containerName="revoke-keys"
Oct 29 22:56:18.757485 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.757395    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.782726 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.782698    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"]
Oct 29 22:56:18.783337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.783314    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a37a29-openshift-default-dockercfg-66nvh\""
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794254    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-credential-socket\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794354    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-data\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794385    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-podinfo\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794447    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-token\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794565    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-certs\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794597    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istiod-ca-cert\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794618    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-socket\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794691    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-envoy\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.795799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.794754    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ws6q\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-kube-api-access-8ws6q\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.914914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.896658    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istiod-ca-cert\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.947042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.947010    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-socket\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.947527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.947495    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-envoy\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.948335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.948313    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ws6q\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-kube-api-access-8ws6q\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.957055 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.957027    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-credential-socket\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.957382 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.957358    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-data\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.958432 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.958409    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-podinfo\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.959022 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.959003    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-token\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.971790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.959442    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-certs\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.972539 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.958344    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-data\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.972733 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.957792    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-credential-socket\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.972860 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.972210    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-envoy\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.972957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.972319    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-certs\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.973062 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.947974    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-socket\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.973217 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.915955    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istiod-ca-cert\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.978408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.978381    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ws6q\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-kube-api-access-8ws6q\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:18.991797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:18.989324    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-podinfo\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:19.088841 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.075824    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-token\") pod \"e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") " pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:19.089110 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.089051    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:19.665809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.665513    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"]
Oct 29 22:56:19.679794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.677835    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:19.679794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.677947    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:19.679794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.677989    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:19.737811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.737730    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"]
Oct 29 22:56:19.764617 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.764583    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"]
Oct 29 22:56:19.765124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.764788    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.915295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.915210    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435e6749-83cf-4b6c-8d54-189ad5aeb204-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.915527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.915502    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v5r4\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-kube-api-access-9v5r4\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.915652 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.915559    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.915699 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.915657    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.915699 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.915679    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.915837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.915699    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.915908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.915821    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.916071 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.916049    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:19.916229 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:19.916213    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.018065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018036    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.018241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018133    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.018241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018165    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.018241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018219    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435e6749-83cf-4b6c-8d54-189ad5aeb204-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.018400 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018246    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v5r4\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-kube-api-access-9v5r4\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.018400 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018282    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.018400 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018379    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.019395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018409    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.019395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018440    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.019395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018598    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.019395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.018727    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.019395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.019142    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.019678 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.019640    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435e6749-83cf-4b6c-8d54-189ad5aeb204-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.019678 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.019637    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.020958 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.020933    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.021459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.021436    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.045896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.045870    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.046007 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.045958    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v5r4\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-kube-api-access-9v5r4\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.093977 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.093946    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:20.393419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.393378    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" event={"ID":"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b","Type":"ContainerStarted","Data":"f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3"}
Oct 29 22:56:20.393589 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.393450    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" event={"ID":"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b","Type":"ContainerStarted","Data":"5535d94cf20daa3dd2cc0d09f2ba81e4147f03b3a49a394eabf946839ef63d17"}
Oct 29 22:56:20.420123 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.420069    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"]
Oct 29 22:56:20.421587 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:20.421554    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435e6749_83cf_4b6c_8d54_189ad5aeb204.slice/crio-b9d162bd0743b060236485f7d003d47e963c3a3319112fb8125a0c4c939f0bdf WatchSource:0}: Error finding container b9d162bd0743b060236485f7d003d47e963c3a3319112fb8125a0c4c939f0bdf: Status 404 returned error can't find the container with id b9d162bd0743b060236485f7d003d47e963c3a3319112fb8125a0c4c939f0bdf
Oct 29 22:56:20.424217 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.424182    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:20.424298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.424252    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:20.424298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.424284    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:20.439802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:20.439717    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podStartSLOduration=2.439700248 podStartE2EDuration="2.439700248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:20.436958451 +0000 UTC m=+816.059869482" watchObservedRunningTime="2026-10-29 22:56:20.439700248 +0000 UTC m=+816.062611271"
Oct 29 22:56:21.090124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.090080    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:21.096466 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.096442    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:21.312574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.312546    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:56:21.638313 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.638273    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" event={"ID":"435e6749-83cf-4b6c-8d54-189ad5aeb204","Type":"ContainerStarted","Data":"13e5ebd47abfca6199c0be3b3ec2aec8a7fcb848b3a4ba7d8cf20cb4e64ef93a"}
Oct 29 22:56:21.638631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.638608    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:21.638772 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.638740    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" event={"ID":"435e6749-83cf-4b6c-8d54-189ad5aeb204","Type":"ContainerStarted","Data":"b9d162bd0743b060236485f7d003d47e963c3a3319112fb8125a0c4c939f0bdf"}
Oct 29 22:56:21.640237 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.640217    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:56:21.693142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:21.693086    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podStartSLOduration=2.693070022 podStartE2EDuration="2.693070022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:21.691216949 +0000 UTC m=+817.314127970" watchObservedRunningTime="2026-10-29 22:56:21.693070022 +0000 UTC m=+817.315981041"
Oct 29 22:56:22.094759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.094675    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:22.096461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.096428    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:56:22.096593 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.096490    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:56:22.833056 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.833019    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"]
Oct 29 22:56:22.837879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.837861    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:22.846741 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.846714    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"]
Oct 29 22:56:22.852328 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.852302    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:22.852436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.852421    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:56:22.900230 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.900198    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q687s\" (UniqueName: \"kubernetes.io/projected/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-kube-api-access-q687s\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:22.900903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.900873    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:22.901101 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:22.901078    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.002964 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.002923    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.003139 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.003037    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.003194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.003161    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q687s\" (UniqueName: \"kubernetes.io/projected/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-kube-api-access-q687s\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.003707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.003684    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.005296 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.005275    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.026815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.026785    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q687s\" (UniqueName: \"kubernetes.io/projected/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-kube-api-access-q687s\") pod \"maas-api-revoke-sub-66b9b771-6vkdz\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.094349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.094269    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:56:23.094349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.094319    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:56:23.154734 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.154691    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:23.832441 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:23.832412    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"]
Oct 29 22:56:23.832709 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:23.832688    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c4ef6a_a9af_4b33_9c43_e1286f8e31ed.slice/crio-c6024e2f046f1558c6774a12a237fe74895233b9eb1604cc5260fa898686a34d WatchSource:0}: Error finding container c6024e2f046f1558c6774a12a237fe74895233b9eb1604cc5260fa898686a34d: Status 404 returned error can't find the container with id c6024e2f046f1558c6774a12a237fe74895233b9eb1604cc5260fa898686a34d
Oct 29 22:56:24.099908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.094519    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:56:24.099908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.094577    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:56:24.833352 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.833312    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz" event={"ID":"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed","Type":"ContainerStarted","Data":"2fbc1d7c29da9f8b71d170c2cb6d746096481d148b825e6804efbb6118daa527"}
Oct 29 22:56:24.833818 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.833397    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz" event={"ID":"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed","Type":"ContainerStarted","Data":"c6024e2f046f1558c6774a12a237fe74895233b9eb1604cc5260fa898686a34d"}
Oct 29 22:56:24.887905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.884748    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz" podStartSLOduration=2.884728786 podStartE2EDuration="2.884728786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:24.881138928 +0000 UTC m=+820.504049949" watchObservedRunningTime="2026-10-29 22:56:24.884728786 +0000 UTC m=+820.507639809"
Oct 29 22:56:24.908751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.908723    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"]
Oct 29 22:56:24.913185 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.913146    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:24.928038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.928012    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a37a29-dockercfg-smvd7\""
Oct 29 22:56:24.928339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.928304    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a37a29\""
Oct 29 22:56:24.933284 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:24.933264    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"]
Oct 29 22:56:25.085843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.080300    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkl9\" (UniqueName: \"kubernetes.io/projected/57d893fd-9b31-4e44-9905-48b6fd168512-kube-api-access-8vkl9\") pod \"payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:25.429114 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.429079    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d893fd-9b31-4e44-9905-48b6fd168512-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:25.429330 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.123526    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:56:25.429409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.429325    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:56:25.521521 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:56:25.521483    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c4ef6a_a9af_4b33_9c43_e1286f8e31ed.slice/crio-conmon-2fbc1d7c29da9f8b71d170c2cb6d746096481d148b825e6804efbb6118daa527.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c4ef6a_a9af_4b33_9c43_e1286f8e31ed.slice/crio-2fbc1d7c29da9f8b71d170c2cb6d746096481d148b825e6804efbb6118daa527.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:56:25.530960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.530931    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkl9\" (UniqueName: \"kubernetes.io/projected/57d893fd-9b31-4e44-9905-48b6fd168512-kube-api-access-8vkl9\") pod \"payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:25.531265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.531229    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d893fd-9b31-4e44-9905-48b6fd168512-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:25.532176 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.532040    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d893fd-9b31-4e44-9905-48b6fd168512-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:25.556658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.556633    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkl9\" (UniqueName: \"kubernetes.io/projected/57d893fd-9b31-4e44-9905-48b6fd168512-kube-api-access-8vkl9\") pod \"payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:25.828027 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.827949    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:25.843886 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.843859    4827 generic.go:356] "Generic (PLEG): container finished" podID="b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" containerID="2fbc1d7c29da9f8b71d170c2cb6d746096481d148b825e6804efbb6118daa527" exitCode=0
Oct 29 22:56:25.844229 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:25.843953    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz" event={"ID":"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed","Type":"ContainerDied","Data":"2fbc1d7c29da9f8b71d170c2cb6d746096481d148b825e6804efbb6118daa527"}
Oct 29 22:56:26.105822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:26.099904    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:56:26.105822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:26.099977    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:56:26.323525 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:26.323493    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d893fd_9b31_4e44_9905_48b6fd168512.slice/crio-8e327662c80c197e6f23061daee69dbbbc7b77f8d7d0c569e384076ea054ef6c WatchSource:0}: Error finding container 8e327662c80c197e6f23061daee69dbbbc7b77f8d7d0c569e384076ea054ef6c: Status 404 returned error can't find the container with id 8e327662c80c197e6f23061daee69dbbbc7b77f8d7d0c569e384076ea054ef6c
Oct 29 22:56:26.324534 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:26.324513    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"]
Oct 29 22:56:27.035039 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.035004    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz" event={"ID":"57d893fd-9b31-4e44-9905-48b6fd168512","Type":"ContainerStarted","Data":"c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425"}
Oct 29 22:56:27.035643 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.035622    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz" event={"ID":"57d893fd-9b31-4e44-9905-48b6fd168512","Type":"ContainerStarted","Data":"8e327662c80c197e6f23061daee69dbbbc7b77f8d7d0c569e384076ea054ef6c"}
Oct 29 22:56:27.035884 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.035868    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:27.081370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.081322    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz" podStartSLOduration=3.081303991 podStartE2EDuration="3.081303991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:27.076994656 +0000 UTC m=+822.699905675" watchObservedRunningTime="2026-10-29 22:56:27.081303991 +0000 UTC m=+822.704215337"
Oct 29 22:56:27.094653 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.094620    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:56:27.094835 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.094676    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:56:27.208740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.208707    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:27.241132 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.241104    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"]
Oct 29 22:56:27.241722 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.241707    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" containerName="revoke-keys"
Oct 29 22:56:27.241793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.241725    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" containerName="revoke-keys"
Oct 29 22:56:27.241907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.241895    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" containerName="revoke-keys"
Oct 29 22:56:27.246086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.246068    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297125    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-openshift-ca\") pod \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") "
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297307    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-kube-api-access-q687s\" (UniqueName: \"kubernetes.io/projected/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-kube-api-access-q687s\") pod \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") "
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297473    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-xks-ca\") pod \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\" (UID: \"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed\") "
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297662    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297758    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kh7s\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-kube-api-access-9kh7s\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297849    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4930d595-577e-4d00-805d-c04ed12523b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297893    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.297936    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4930d595-577e-4d00-805d-c04ed12523b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.298011    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.298038    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.298133    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.298186    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.310852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.298615    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-openshift-ca" pod "b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" (UID: "b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411032    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4930d595-577e-4d00-805d-c04ed12523b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411101    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411151    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4930d595-577e-4d00-805d-c04ed12523b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411236    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411265    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411369    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411425    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411465    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411594    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kh7s\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-kube-api-access-9kh7s\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.415861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.411695    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:27.462792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.422271    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.462792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.422579    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.462792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.423242    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4930d595-577e-4d00-805d-c04ed12523b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.462792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.423468    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.462792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.440186    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.462792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.445877    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"]
Oct 29 22:56:27.492907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.492668    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kh7s\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-kube-api-access-9kh7s\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.510697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.494971    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4930d595-577e-4d00-805d-c04ed12523b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.565579 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.565415    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.591335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.591266    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-kube-api-access-q687s" pod "b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" (UID: "b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed"). InnerVolumeSpecName "kube-api-access-q687s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:27.614865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.613980    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-xks-ca" pod "b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" (UID: "b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:27.624824 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.619350    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fg5ws\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:27.642702 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.642652    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q687s\" (UniqueName: \"kubernetes.io/projected/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-kube-api-access-q687s\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:27.642702 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.642687    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:27.868093 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:27.868013    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:28.097662 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.097596    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:56:28.098146 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.097657    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:56:28.098966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.098864    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz" event={"ID":"b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed","Type":"ContainerDied","Data":"c6024e2f046f1558c6774a12a237fe74895233b9eb1604cc5260fa898686a34d"}
Oct 29 22:56:28.098966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.098921    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6024e2f046f1558c6774a12a237fe74895233b9eb1604cc5260fa898686a34d"
Oct 29 22:56:28.099128 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.099012    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"
Oct 29 22:56:28.321032 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:28.321000    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4930d595_577e_4d00_805d_c04ed12523b6.slice/crio-845c4aa327938f7d0891f01dac1535b9d40a64f0fc5445b8298114bef51e4c9d WatchSource:0}: Error finding container 845c4aa327938f7d0891f01dac1535b9d40a64f0fc5445b8298114bef51e4c9d: Status 404 returned error can't find the container with id 845c4aa327938f7d0891f01dac1535b9d40a64f0fc5445b8298114bef51e4c9d
Oct 29 22:56:28.321164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.321086    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"]
Oct 29 22:56:28.323617 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.323575    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:28.323727 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.323655    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:28.323727 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:28.323694    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:29.099494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.099468    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:29.100007 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.099873    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:29.100509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.100487    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 22:56:29.107547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.107524    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" event={"ID":"4930d595-577e-4d00-805d-c04ed12523b6","Type":"ContainerStarted","Data":"f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2"}
Oct 29 22:56:29.107700 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.107567    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" event={"ID":"4930d595-577e-4d00-805d-c04ed12523b6","Type":"ContainerStarted","Data":"845c4aa327938f7d0891f01dac1535b9d40a64f0fc5445b8298114bef51e4c9d"}
Oct 29 22:56:29.252969 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.252918    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podStartSLOduration=2.252902355 podStartE2EDuration="2.252902355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:29.248395443 +0000 UTC m=+824.871306464" watchObservedRunningTime="2026-10-29 22:56:29.252902355 +0000 UTC m=+824.875813433"
Oct 29 22:56:29.877791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.872156    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:29.886791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.879886    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:29.886791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:29.879943    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:30.886222 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:30.886185    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:30.886817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:30.886760    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:31.885794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:31.882150    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:31.885794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:31.882250    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:32.035635 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.035605    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"]
Oct 29 22:56:32.040122 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.040104    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.059575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.059551    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"]
Oct 29 22:56:32.090055 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.090027    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:56:32.090590 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.090569    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:32.141354 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.141273    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4v4\" (UniqueName: \"kubernetes.io/projected/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-kube-api-access-8l4v4\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.141544 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.141515    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.141607 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.141568    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.242847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.242812    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4v4\" (UniqueName: \"kubernetes.io/projected/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-kube-api-access-8l4v4\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.243059 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.243041    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.243120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.243106    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.243848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.243830    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.245504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.245485    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.290076 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.290047    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4v4\" (UniqueName: \"kubernetes.io/projected/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-kube-api-access-8l4v4\") pod \"maas-api-revoke-sub-f7616a32-6qhqg\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.354299 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.354264    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:32.692943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.692846    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"]
Oct 29 22:56:32.698336 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:32.698304    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0d208f_c1ba_48e2_8fa1_9b51a9c651f3.slice/crio-2baae2bf806853c6e98d1bc543f9ab3414087842b7d88b3f70d615cc33f35101 WatchSource:0}: Error finding container 2baae2bf806853c6e98d1bc543f9ab3414087842b7d88b3f70d615cc33f35101: Status 404 returned error can't find the container with id 2baae2bf806853c6e98d1bc543f9ab3414087842b7d88b3f70d615cc33f35101
Oct 29 22:56:32.880799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.880333    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:32.880799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:32.880396    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:32.923390 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:56:32.923343    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:56:33.158784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:33.157927    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg" event={"ID":"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3","Type":"ContainerStarted","Data":"2baae2bf806853c6e98d1bc543f9ab3414087842b7d88b3f70d615cc33f35101"}
Oct 29 22:56:33.877795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:33.876874    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:33.877795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:33.876942    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:34.245716 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:34.244213    4827 generic.go:356] "Generic (PLEG): container finished" podID="2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" containerID="d882bf8b088022a8cd04e8b03614782454b7310c65d3e699f7a5c4426be75f87" exitCode=0
Oct 29 22:56:34.245716 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:34.244314    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg" event={"ID":"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3","Type":"ContainerDied","Data":"d882bf8b088022a8cd04e8b03614782454b7310c65d3e699f7a5c4426be75f87"}
Oct 29 22:56:34.888808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:34.849461    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"]
Oct 29 22:56:34.912844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:34.891242    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:34.912844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:34.891301    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:35.080784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.067162    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"]
Oct 29 22:56:35.080784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.067304    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.124377 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.124319    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:35.125029 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.124997    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:35.157833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.157803    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2wf\" (UniqueName: \"kubernetes.io/projected/5f75a5ce-e2a9-407a-b399-937dd1765358-kube-api-access-mv2wf\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.158339 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.158323    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.158685 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.158658    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.313792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.306951    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2wf\" (UniqueName: \"kubernetes.io/projected/5f75a5ce-e2a9-407a-b399-937dd1765358-kube-api-access-mv2wf\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.313792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.307052    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.313792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.307097    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.313792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.308145    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.329816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.323397    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.379707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.379670    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2wf\" (UniqueName: \"kubernetes.io/projected/5f75a5ce-e2a9-407a-b399-937dd1765358-kube-api-access-mv2wf\") pod \"maas-api-revoke-sub-3413b987-46s9m\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.459047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.458925    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:35.508256 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.508203    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:35.707553 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.707490    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:35.726859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.720222    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:35.776806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.774101    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:35.776806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.774654    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:35.886881 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.884605    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:35.886881 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:35.884665    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:36.095844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.090262    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:36.129789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.097857    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:36.185797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.160544    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:36.276565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.187109    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:36.277166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.267921    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:56:36.303965 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.303935    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:56:36.480702 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.480657    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:36.571818 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.548324    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:36.614788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.574755    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:36.745848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.715171    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:36.773794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.719269    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:36.774109 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.721399    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:36.813036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.799041    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:36.813036 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.798824    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:36.963790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.882512    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:56:36.963790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.882561    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"]
Oct 29 22:56:36.963790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.961303    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:36.963790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:36.961356    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:37.095492 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.095455    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:37.110814 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.110746    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:37.269342 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.156590    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f" podUID="89beb379-c7e3-441a-90cb-5572b7d934da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.24:8000/ready\": context deadline exceeded"
Oct 29 22:56:37.274911 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.256122    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:37.280987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.280945    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:37.453810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.420563    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:56:37.453810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.420717    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:56:37.453810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.420754    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:56:37.453810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.421324    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:56:37.453810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.421462    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:56:37.453810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.422816    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:37.547413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.547367    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:37.601791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.598455    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.042870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.030409    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.042870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.030725    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.364850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.258122    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.364850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.258291    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgwh\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-kube-api-access-skgwh\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.364850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.258348    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.364850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.258677    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad67055-0ab0-4997-b160-499db019ba1b-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.364850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.284891    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.463791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.185989    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.463791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.186144    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.463791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:37.605913    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"]
Oct 29 22:56:38.463791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.098668    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Oct 29 22:56:38.463791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.098632    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.463791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.098830    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:38.463791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.172030    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:38.557111 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.557060    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.583679 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.579602    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.600793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.596826    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.631850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.507112    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.666788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.664789    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:38.721887 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.721832    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:38.722232 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.722203    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.722375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.722352    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 29 22:56:38.852439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.852388    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.873889 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.853122    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.933871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.931405    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad67055-0ab0-4997-b160-499db019ba1b-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:38.934050 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.933877    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:38.961830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.961143    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:38.961830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:38.961199    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:39.056879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.056835    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082078    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082158    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082195    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082237    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgwh\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-kube-api-access-skgwh\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082276    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082326    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad67055-0ab0-4997-b160-499db019ba1b-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082455    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad67055-0ab0-4997-b160-499db019ba1b-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.082676    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.083407    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.084826    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad67055-0ab0-4997-b160-499db019ba1b-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.085075    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.085099    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.085437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.085336    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.311044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.309353    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.334902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.327309    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad67055-0ab0-4997-b160-499db019ba1b-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.397850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.352818    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:39.397850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.354007    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.397850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.374493    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgwh\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-kube-api-access-skgwh\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.454639 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.454602    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-xks-ca\") pod \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") "
Oct 29 22:56:39.473161 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.455500    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.473161 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.455730    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.476800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.473569    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:39.490709 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.490666    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.491286 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.491256    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.554244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.491936    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.554244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.537791    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"]
Oct 29 22:56:39.554244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.537881    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg" event={"ID":"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3","Type":"ContainerDied","Data":"2baae2bf806853c6e98d1bc543f9ab3414087842b7d88b3f70d615cc33f35101"}
Oct 29 22:56:39.554244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.537917    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2baae2bf806853c6e98d1bc543f9ab3414087842b7d88b3f70d615cc33f35101"
Oct 29 22:56:39.554244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.538051    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:39.554244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.538089    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:56:39.554244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.538126    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:56:39.598487 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.598403    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-kube-api-access-8l4v4\" (UniqueName: \"kubernetes.io/projected/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-kube-api-access-8l4v4\") pod \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") "
Oct 29 22:56:39.637271 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.637238    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-openshift-ca\") pod \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\" (UID: \"2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3\") "
Oct 29 22:56:39.652926 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.625921    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.653321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.625953    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.654073 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.625883    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.654920 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.654883    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.655832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.655809    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.659871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.659839    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.660092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.660070    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.663786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.660296    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.664103 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.664083    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjftw\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-kube-api-access-gjftw\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.667524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.667081    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-openshift-ca" pod "2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" (UID: "2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:39.667524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.667151    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:39.681800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.673947    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.705864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.698803    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.705864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.698930    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.705864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.701090    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:39.774519 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.774487    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"]
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.812309    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjftw\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-kube-api-access-gjftw\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927029    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927295    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927340    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927446    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927504    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927529    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927590    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927651    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:39.938859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.927804    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:39.939949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.883275    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:39.939949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.939568    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:39.939949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.881869    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.899257    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-xks-ca" pod "2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" (UID: "2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.962994    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.881913    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.964701    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-kube-api-access-8l4v4" pod "2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" (UID: "2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3"). InnerVolumeSpecName "kube-api-access-8l4v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.965445    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.965959    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.966211    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.969954    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.991551    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:40.018078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:39.997823    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.063189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.037099    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:40.063189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.037135    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4v4\" (UniqueName: \"kubernetes.io/projected/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3-kube-api-access-8l4v4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:40.287852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.113959    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 29 22:56:40.372988 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.372882    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:40.372988 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.372938    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.433826 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.366632    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.649956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.529932    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:40.649956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.551735    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:40.649956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.555178    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.798806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.658991    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.798806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.659418    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:40.798806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.659687    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:40.798806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.681735    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:40.798806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.686830    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:40.798806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.715464    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.925810    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.925877    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.933314    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjftw\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-kube-api-access-gjftw\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.939856    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.940076    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.963343    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.964118    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:41.055792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:40.964684    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m" event={"ID":"5f75a5ce-e2a9-407a-b399-937dd1765358","Type":"ContainerStarted","Data":"8a84a479f67e38a91e960af70e015e7967fc58f9b690fdf910fd665b8af408a9"}
Oct 29 22:56:41.270130 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.270032    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:41.474794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.433228    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:41.568192 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.568081    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:41.614871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.584920    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:41.658817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.597038    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:41.658817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.647399    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"]
Oct 29 22:56:41.659701 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.659654    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:41.704790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.661059    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:41.705463 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.661119    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:41.873046 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.873000    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:56:41.887789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.885924    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:41.887789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.885996    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:41.925155 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.925120    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:41.951395 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.951355    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:41.951685 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.950519    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:41.968046 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:41.967919    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:42.012175 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.007198    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:42.057555 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.057512    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:42.442812 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.433897    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:42.646817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.593851    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:42.646817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.616358    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:42.646817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.633637    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:42.646817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.645365    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:42.701840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.699087    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:42.820481 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.710860    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:42.820481 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.710941    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:42.900316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.900266    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:42.948790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:42.923101    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:43.019002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.018903    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.021974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.021937    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.112870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.112833    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:43.113114 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.113081    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:43.161806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.128117    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.161806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.128321    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.161806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.152243    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"]
Oct 29 22:56:43.161806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.153279    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" containerName="revoke-keys"
Oct 29 22:56:43.161806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.153302    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" containerName="revoke-keys"
Oct 29 22:56:43.161806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.153520    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" containerName="revoke-keys"
Oct 29 22:56:43.220833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.220073    4827 generic.go:356] "Generic (PLEG): container finished" podID="5f75a5ce-e2a9-407a-b399-937dd1765358" containerID="b569fde66eef328e1206ddd76336a56cd981ff8b874b9d5fabff220dcd3b4b69" exitCode=0
Oct 29 22:56:43.297207 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.297125    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m" event={"ID":"5f75a5ce-e2a9-407a-b399-937dd1765358","Type":"ContainerDied","Data":"b569fde66eef328e1206ddd76336a56cd981ff8b874b9d5fabff220dcd3b4b69"}
Oct 29 22:56:43.297524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.297501    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.354939 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.354905    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.355194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.355175    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.355298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.355285    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.355416 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.355404    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.355631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.355615    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.355811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.355794    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.361954 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.361920    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.362254 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.362234    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4025821f-f08b-41af-9cd0-45a8b54f5d78-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.362383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.362367    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxjc\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-kube-api-access-fzxjc\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.427134 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.427096    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"]
Oct 29 22:56:43.449065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.449016    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.527562    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.527644    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.527686    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.527729    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.527892    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.527978    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.528053    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.528173    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4025821f-f08b-41af-9cd0-45a8b54f5d78-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.566852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.528212    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxjc\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-kube-api-access-fzxjc\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.613415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.599046    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.613415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.599309    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.613415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.599519    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.613415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.599788    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.613415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.600351    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4025821f-f08b-41af-9cd0-45a8b54f5d78-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.613415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.602100    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:43.627031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.618013    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.661940 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.635870    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:43.665433 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.662580    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.769792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.740209    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:43.769792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.740566    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:43.769792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.743587    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:43.861265 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:56:43.846439    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice/crio-111ff9556758f534b5af86f13bb6b32be314d7c01b1b559d4b36f3236e0eb5f1 WatchSource:0}: Error finding container 111ff9556758f534b5af86f13bb6b32be314d7c01b1b559d4b36f3236e0eb5f1: Status 404 returned error can't find the container with id 111ff9556758f534b5af86f13bb6b32be314d7c01b1b559d4b36f3236e0eb5f1
Oct 29 22:56:43.939690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.939648    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:56:43.940560 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.940535    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:56:43.941402 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:43.941374    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:56:44.014793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.011063    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:44.058759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.031370    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:44.061530 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.060830    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.267117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.130683    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.267117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.146656    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:44.267117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.146756    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:44.267117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.146822    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:44.267117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.147921    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxjc\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-kube-api-access-fzxjc\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:44.267117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.156302    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:44.267117 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.185516    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:44.342832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.335140    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:44.717888 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.717841    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:44.790347 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.736593    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.814793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.814529    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.814971 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.814941    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.915075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.822088    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.915075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.822154    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.915075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.822202    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:44.942952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:44.942855    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:45.034413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:45.034374    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:45.063914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:45.063864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:45.086616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:45.078483    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" event={"ID":"1ad67055-0ab0-4997-b160-499db019ba1b","Type":"ContainerStarted","Data":"111ff9556758f534b5af86f13bb6b32be314d7c01b1b559d4b36f3236e0eb5f1"}
Oct 29 22:56:45.153573 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:45.131008    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:46.115286 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.092255    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.116872    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.133404    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.155154    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.155917    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.156607    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.156921    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.157201    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.159151    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:46.182346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.177999    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:46.225825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.214111    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:46.225825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.214179    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:46.253626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.253571    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:46.256966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.256928    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:46.257186 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.257163    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:46.779880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:46.779168    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:47.189165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.188906    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:47.190955 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.190922    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:47.191552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.191526    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:47.192078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.192045    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:47.192570 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.192540    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:47.194172 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.194148    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:47.315056 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.315009    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:47.468436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.315380    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:47.473974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.319836    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:47.474443 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.319890    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:47.474907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.325296    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:47.475098 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.475063    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:47.685283 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.685204    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:47.685596 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.685547    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:47.685811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.685788    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:56:47.697366 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:56:47.697333    4827 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.043s"
Oct 29 22:56:47.697690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.697577    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" event={"ID":"9f572783-4d2e-47bd-a5e2-543d7d1526a5","Type":"ContainerStarted","Data":"bb19c5c4451536ae2be99090a03648643293288a65b92179d16b2eb8311698f6"}
Oct 29 22:56:47.697903 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.697879    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:56:47.756800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.753210    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:47.768167 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.768137    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 22:56:47.876681 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.876647    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Oct 29 22:56:48.106914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.955979    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:48.106914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:47.973354    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 29 22:56:48.198921 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.192878    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:56:48.216891 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.215907    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:48.216891 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.216557    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:48.216891 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.216832    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:48.269990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.217363    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:48.277095 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.277056    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:48.364862 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.357344    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Oct 29 22:56:48.386797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.384862    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:48.447789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.419444    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:48.485788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.475085    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:48.485788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.480952    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:48.509221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.509192    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:56:48.629656 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.524156    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-openshift-ca\") pod \"5f75a5ce-e2a9-407a-b399-937dd1765358\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") "
Oct 29 22:56:48.734795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.681706    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-xks-ca\") pod \"5f75a5ce-e2a9-407a-b399-937dd1765358\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") "
Oct 29 22:56:48.734795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.681906    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f75a5ce-e2a9-407a-b399-937dd1765358-kube-api-access-mv2wf\" (UniqueName: \"kubernetes.io/projected/5f75a5ce-e2a9-407a-b399-937dd1765358-kube-api-access-mv2wf\") pod \"5f75a5ce-e2a9-407a-b399-937dd1765358\" (UID: \"5f75a5ce-e2a9-407a-b399-937dd1765358\") "
Oct 29 22:56:48.734795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.538951    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-openshift-ca" pod "5f75a5ce-e2a9-407a-b399-937dd1765358" (UID: "5f75a5ce-e2a9-407a-b399-937dd1765358"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:48.734795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.731461    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 29 22:56:48.819803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:48.805660    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:49.278375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.001157    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:49.278375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.001254    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 29 22:56:49.278375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.001291    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 29 22:56:49.337803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.334391    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:49.343904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.343863    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:49.351151 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.345933    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:49.474793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.380904    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:49.474793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.384262    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:49.500790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.475561    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:49.734790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.710106    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.016928 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.783595    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f75a5ce-e2a9-407a-b399-937dd1765358-kube-api-access-mv2wf" pod "5f75a5ce-e2a9-407a-b399-937dd1765358" (UID: "5f75a5ce-e2a9-407a-b399-937dd1765358"). InnerVolumeSpecName "kube-api-access-mv2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:50.016928 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.787385    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:56:49.923339    4827 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.412s"
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030600    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"]
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030707    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030749    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030798    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m" event={"ID":"5f75a5ce-e2a9-407a-b399-937dd1765358","Type":"ContainerDied","Data":"8a84a479f67e38a91e960af70e015e7967fc58f9b690fdf910fd665b8af408a9"}
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030843    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a84a479f67e38a91e960af70e015e7967fc58f9b690fdf910fd665b8af408a9"
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030867    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" event={"ID":"4025821f-f08b-41af-9cd0-45a8b54f5d78","Type":"ContainerStarted","Data":"d03bb442b265fffbfff4e6605bca80aded046cc97175eafdd9fd5a27c0e9833f"}
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030957    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"]
Oct 29 22:56:50.034902 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.030989    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"]
Oct 29 22:56:50.058788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.938540    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded"
Oct 29 22:56:50.058788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:49.969912    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.087950 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.084036    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.113799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.113508    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2wf\" (UniqueName: \"kubernetes.io/projected/5f75a5ce-e2a9-407a-b399-937dd1765358-kube-api-access-mv2wf\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:50.126161 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.126101    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.153251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.153208    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.233035 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.232996    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" event={"ID":"1ad67055-0ab0-4997-b160-499db019ba1b","Type":"ContainerStarted","Data":"26afd1d7da919b9e0a316d2788c6ba9a07361a0eb3d234da0d31c59585ec3043"}
Oct 29 22:56:50.267320 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.267289    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:50.337787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.334555    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.364989 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.364910    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.365433 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.365409    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:50.394173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.394139    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:50.543790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.516951    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-xks-ca" pod "5f75a5ce-e2a9-407a-b399-937dd1765358" (UID: "5f75a5ce-e2a9-407a-b399-937dd1765358"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:50.543790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.518710    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f75a5ce-e2a9-407a-b399-937dd1765358-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:56:50.631873 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.621979    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.631873 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.622322    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:50.779974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.767881    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:50.779974 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:50.767943    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:51.104239 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.086714    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.128505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.117700    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:56:51.128505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.117752    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:51.227349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.205713    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.227349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.206678    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:51.227349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.207440    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.241446 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.240562    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.417015 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.416975    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:56:51.651300 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.420353    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.652859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.420540    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.654095 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.420739    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.916407 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.916320    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:51.916677 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.916648    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:51.952177 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.930877    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:52.146788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.122565    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:52.146788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.123715    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:52.155794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.952632    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:52.215096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.214978    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.274213 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.274134    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.303789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:51.933513    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:52.303789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.292563    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:56:52.303789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.292626    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:56:52.303789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.292683    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:56:52.322189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.316493    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:56:52.444033 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.443988    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:56:52.846796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.833198    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:52.846796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.833258    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:53.013791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.012598    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.148543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:52.967182    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.148543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.077150    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:56:53.148543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.077213    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:56:53.281817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.184865    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.281817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.215865    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.281817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.216803    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:53.313799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.300069    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:53.696877 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.301044    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.700519 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.700475    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.701139 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.599698    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:56:53.701325 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.701297    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:56:53.722921 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.301077    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:56:53.750276 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:56:53.728942    4827 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.18s"
Oct 29 22:56:53.750276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.728989    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:53.750276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.729087    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" event={"ID":"9f572783-4d2e-47bd-a5e2-543d7d1526a5","Type":"ContainerStarted","Data":"2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd"}
Oct 29 22:56:53.750276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.729118    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:53.750276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.729177    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:56:53.750276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.729214    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:53.771900 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.763884    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.861823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.819677    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:53.861823 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:53.819735    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.099215    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.100072    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.148034    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.157060    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.157110    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.219888    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.220296    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.220638    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:56:54.302856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.289334    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podStartSLOduration=18.289310483 podStartE2EDuration="18.289310483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:50.632518234 +0000 UTC m=+846.255429257" watchObservedRunningTime="2026-10-29 22:56:54.289310483 +0000 UTC m=+849.912221504"
Oct 29 22:56:54.402795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.362261    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:56:54.402795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.362321    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:56:54.695013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.694968    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:54.759286 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.703057    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.760743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.704930    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.798518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.762527    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:54.798518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.762573    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:54.798518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.783872    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:56:54.808814 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.800045    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.848182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.848154    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:56:54.848434 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.848417    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" event={"ID":"4025821f-f08b-41af-9cd0-45a8b54f5d78","Type":"ContainerStarted","Data":"0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75"}
Oct 29 22:56:54.848605 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.848594    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:56:54.848743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.848735    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:56:54.969570 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:54.968905    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:55.105803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.102313    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:55.150065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.149218    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:55.359110 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.359019    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:56:55.409792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.387114    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:56:55.463390 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.438346    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:56:55.463390 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.411740    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:55.791857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.781640    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:56.006797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:55.990279    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.008371 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.008331    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.039719 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.039676    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.068849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.057057    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:56.255794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.059699    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.264794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.109985    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podStartSLOduration=20.109955067 podStartE2EDuration="20.109955067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:52.992706445 +0000 UTC m=+848.615617466" watchObservedRunningTime="2026-10-29 22:56:56.109955067 +0000 UTC m=+851.732866088"
Oct 29 22:56:56.349842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.121153    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:56:56.350017 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.349891    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:56:56.353323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.121956    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.359337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.251924    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.403144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.403097    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:56.404258 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.404230    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:56:56.908951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.496216    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.948076 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.934298    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:56.973229 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:56.959196    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:56:57.063165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.063122    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:57.169868 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.066915    4827 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:56:57.169868 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.100801    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:56:57.173155 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.173127    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.173360 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.173336    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:57.229821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.212604    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.229821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.212691    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:56:57.229821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.213297    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.229821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.213334    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:56:57.251506 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.100858    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:57.295931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.253805    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:56:57.295931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.278008    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"]
Oct 29 22:56:57.295931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.278196    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bd927-pwqql"]
Oct 29 22:56:57.295931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.263590    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:57.295931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.263646    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:57.344988 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.344933    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.345193 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.345010    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:56:57.379788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.371519    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:57.379788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.371985    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:56:57.453914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.430570    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:57.485037 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.475202    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:57.764840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.764401    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.764840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.764476    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:58.000794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:57.986891    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.018965 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.004343    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:58.056998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.020870    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.056998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.020911    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:56:58.106979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.099509    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Oct 29 22:56:58.231923 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.231870    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.275839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.242102    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.275839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.273535    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.345793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.320884    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.373993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.361227    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:56:58.373993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.370167    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:56:58.410222 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.410173    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:56:58.417896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.417854    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:56:58.418218 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.418198    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:56:58.418351 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.418333    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:56:58.434641 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.419286    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.501254 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.460395    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:56:58.501254 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.493869    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x" podUID="f90fd47c-4d11-4faa-8eb9-d394960cc2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.509842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.509794    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.547968 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.547881    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f" podUID="89beb379-c7e3-441a-90cb-5572b7d934da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.24:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.564988 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.564948    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:58.707980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.670991    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 29 22:56:58.729333 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.729147    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2ac21b-3f72-42d7-88e3-5cbe0f5f9627" path="/var/lib/kubelet/pods/db2ac21b-3f72-42d7-88e3-5cbe0f5f9627/volumes"
Oct 29 22:56:58.730005 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.729979    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"]
Oct 29 22:56:58.731035 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.731018    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5f75a5ce-e2a9-407a-b399-937dd1765358" containerName="revoke-keys"
Oct 29 22:56:58.731157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.731145    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f75a5ce-e2a9-407a-b399-937dd1765358" containerName="revoke-keys"
Oct 29 22:56:58.731482 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.731472    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f75a5ce-e2a9-407a-b399-937dd1765358" containerName="revoke-keys"
Oct 29 22:56:58.741107 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.737269    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:58.741211 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.741133    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:58.749371 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.749349    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:56:58.750310 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.750290    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:58.751924 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.751907    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:56:58.774792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.757324    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:56:58.775333 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.775310    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:58.776541 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.776512    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:56:58.992804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.990151    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crs6w\" (UniqueName: \"kubernetes.io/projected/1dd433c4-f254-4087-a651-74ef9cc12d96-kube-api-access-crs6w\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:58.992804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.990333    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:58.992804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.990386    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:58.992804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:58.990921    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:56:59.013456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.011081    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:56:59.013456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.011132    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:56:59.028149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.026754    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:56:59.028149 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.027309    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:56:59.099844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.074900    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:59.444912 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.096743    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crs6w\" (UniqueName: \"kubernetes.io/projected/1dd433c4-f254-4087-a651-74ef9cc12d96-kube-api-access-crs6w\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:59.445667 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.445639    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:59.446688 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.446656    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:59.609120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.255514    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podStartSLOduration=21.255494523 podStartE2EDuration="21.255494523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:53.914429791 +0000 UTC m=+849.537340812" watchObservedRunningTime="2026-10-29 22:56:59.255494523 +0000 UTC m=+854.878405544"
Oct 29 22:56:59.740843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.343565    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:56:59.740843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.729144    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:56:59.740843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.122824    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:56:59.740843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.729486    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"]
Oct 29 22:56:59.740843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.729644    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:56:59.740843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.729675    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:56:59.740843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.363164    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:59.752716 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.752678    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:56:59.922791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.877055    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:56:59.972842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.221132    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crs6w\" (UniqueName: \"kubernetes.io/projected/1dd433c4-f254-4087-a651-74ef9cc12d96-kube-api-access-crs6w\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:59.972842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.446554    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:56:59.972842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:56:59.898621    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e1dc368-mfjc5\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:57:00.046532 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.043465    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:57:00.049811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.049783    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:57:00.050040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.050016    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:57:00.148788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.140479    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:57:00.492750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.492714    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:57:00.493094 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.493062    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:57:00.514462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.514433    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"]
Oct 29 22:57:00.538555 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.538526    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"]
Oct 29 22:57:00.538888 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.538865    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.561176 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.561151    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-27d16a-openshift-default-dockercfg-dq7sj\""
Oct 29 22:57:00.580887 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.580841    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"]
Oct 29 22:57:00.594138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.592099    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"]
Oct 29 22:57:00.594138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.592213    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.602966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.598236    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"]
Oct 29 22:57:00.603451 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.603417    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8923c498-8a01-4288-85ae-8f71b92545bf-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.603553 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.603494    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.603626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.603559    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.603626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.603611    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.603949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.603925    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67lp\" (UniqueName: \"kubernetes.io/projected/a7304cb6-90c9-45c0-b6a4-5caec95733a6-kube-api-access-f67lp\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.604014 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.603981    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.604105 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.604089    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.604195 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.604179    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdjf4\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-kube-api-access-gdjf4\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.604326 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.604305    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.632587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.608349    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.632587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.608392    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8923c498-8a01-4288-85ae-8f71b92545bf-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.632587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.608421    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.690860 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.689538    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"]
Oct 29 22:57:00.690860 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.689603    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"]
Oct 29 22:57:00.694082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.691326    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.712706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.711167    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.712706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.712674    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.714784 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.713525    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.719163    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdjf4\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-kube-api-access-gdjf4\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.719557    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.720164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720010    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720052    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8923c498-8a01-4288-85ae-8f71b92545bf-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720078    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720129    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8923c498-8a01-4288-85ae-8f71b92545bf-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720164 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720169    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.720614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720244    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720293    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.720614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.720391    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67lp\" (UniqueName: \"kubernetes.io/projected/a7304cb6-90c9-45c0-b6a4-5caec95733a6-kube-api-access-f67lp\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.736626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.725268    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.736626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.725963    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.736626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.726500    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8923c498-8a01-4288-85ae-8f71b92545bf-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.736626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.726727    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.736626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.726954    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.736626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.730265    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8923c498-8a01-4288-85ae-8f71b92545bf-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.736626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.734268    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.738505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.738461    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.747386 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.747342    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67lp\" (UniqueName: \"kubernetes.io/projected/a7304cb6-90c9-45c0-b6a4-5caec95733a6-kube-api-access-f67lp\") pod \"maas-api-revoke-sub-8e2a1a26-wwqq6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.805895 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.805826    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.811865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.811731    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdjf4\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-kube-api-access-gdjf4\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-5p8ld\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822476    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d297af-035a-421e-bf23-38d878371101-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822542    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfnz\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-kube-api-access-vxfnz\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822599    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d297af-035a-421e-bf23-38d878371101-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822662    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822709    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822832    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822870    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.822895    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.823574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.823099    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.825267 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.825235    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"]
Oct 29 22:57:00.825416 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.825399    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.842027 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.841631    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cd525f-openshift-default-dockercfg-rkhnr\""
Oct 29 22:57:00.876977 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.872534    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:00.929166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929127    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.929300 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929198    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.929300 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929253    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.929419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929308    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.929419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929344    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.929419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929397    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9th\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-kube-api-access-gh9th\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.929565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929472    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c197150f-64b3-41b9-a0ed-951e3cb3212d-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.929624 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929588    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.929677 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929621    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.929728 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.929690    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.930502 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.930475    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.930696 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.930674    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d297af-035a-421e-bf23-38d878371101-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.930905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.930885    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfnz\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-kube-api-access-vxfnz\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.931198 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.931178    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.932868 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.932847    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.933331 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.933137    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d297af-035a-421e-bf23-38d878371101-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.933527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.933511    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.933658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.933644    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:00.933799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.933786    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.934413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.934392    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.935125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.935101    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.935364 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.935345    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.944207 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.944074    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d297af-035a-421e-bf23-38d878371101-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.944343 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.944305    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:00.967390 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.967366    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfnz\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-kube-api-access-vxfnz\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.968251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.967877    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.980421 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.978829    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"]
Oct 29 22:57:00.980421 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.978982    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:57:00.984087 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.984027    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 22:57:00.991079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.991050    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:00.991253 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:00.991232    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d297af-035a-421e-bf23-38d878371101-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:01.010203 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.009139    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:57:01.010203 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.009196    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:57:01.021409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.021378    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:01.048397 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.048333    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.054775    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.054845    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9th\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-kube-api-access-gh9th\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.054928    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c197150f-64b3-41b9-a0ed-951e3cb3212d-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.055058    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.055088    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.055235    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.055315    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.056552 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.055959    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c197150f-64b3-41b9-a0ed-951e3cb3212d-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.058311 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.058126    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.059390 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.058440    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.063462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.063434    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.063751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.063731    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.064033 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.064009    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.157840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.157735    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.175265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.169061    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.175265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.169315    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9th\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-kube-api-access-gh9th\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.176565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.176518    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.176985 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.176960    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5" event={"ID":"1dd433c4-f254-4087-a651-74ef9cc12d96","Type":"ContainerStarted","Data":"d6363bccfe6f27c650cd802e342b07475289a90f0bbb57e14e11b1d7244b3d1b"}
Oct 29 22:57:01.309743 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:57:01.309239    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8923c498_8a01_4288_85ae_8f71b92545bf.slice/crio-340f3c2b38e448f7a4a41edb513421eb1977c473f3047694a381b66708322241 WatchSource:0}: Error finding container 340f3c2b38e448f7a4a41edb513421eb1977c473f3047694a381b66708322241: Status 404 returned error can't find the container with id 340f3c2b38e448f7a4a41edb513421eb1977c473f3047694a381b66708322241
Oct 29 22:57:01.316217 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.316161    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:01.316463 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.316436    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:01.316652 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.316631    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:01.341604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.341235    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:57:01.341604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.341282    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 29 22:57:01.344941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.344914    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"]
Oct 29 22:57:01.493849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.470485    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:01.723061 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.721214    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"]
Oct 29 22:57:01.911743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.908865    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"]
Oct 29 22:57:01.979141 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.977950    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"]
Oct 29 22:57:01.979141 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.978450    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:01.979141 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.978522    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:01.979141 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:01.978563    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:02.008099 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.006940    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:02.008099 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.007049    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:02.008099 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.007098    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:02.203483 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.202384    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5" event={"ID":"1dd433c4-f254-4087-a651-74ef9cc12d96","Type":"ContainerStarted","Data":"fb947ac7c49708d9166c7d556175d8119b4cf7789f6ddbc807309989a31c73e9"}
Oct 29 22:57:02.205465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.205396    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" event={"ID":"c197150f-64b3-41b9-a0ed-951e3cb3212d","Type":"ContainerStarted","Data":"6e7fbc85e45f8845a74c8aa99ccd759f3338ac3034428546a6958cf665f08447"}
Oct 29 22:57:02.208599 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.207901    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:02.208599 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.207948    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:02.211723 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.211033    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" event={"ID":"8923c498-8a01-4288-85ae-8f71b92545bf","Type":"ContainerStarted","Data":"6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a"}
Oct 29 22:57:02.211723 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.211080    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" event={"ID":"8923c498-8a01-4288-85ae-8f71b92545bf","Type":"ContainerStarted","Data":"340f3c2b38e448f7a4a41edb513421eb1977c473f3047694a381b66708322241"}
Oct 29 22:57:02.213563 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.213499    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" event={"ID":"61d297af-035a-421e-bf23-38d878371101","Type":"ContainerStarted","Data":"a15697f266ade7a79fd1fe01ed929da52a31ba2d24647e262d8acba5bb661ecc"}
Oct 29 22:57:02.215317 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.215259    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6" event={"ID":"a7304cb6-90c9-45c0-b6a4-5caec95733a6","Type":"ContainerStarted","Data":"04d13ec491d0d082694c3fce1742ac6a21714a27477886d05eb79d2d2a63b518"}
Oct 29 22:57:02.256787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.255654    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5" podStartSLOduration=9.255635669 podStartE2EDuration="9.255635669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:02.246816584 +0000 UTC m=+857.869727605" watchObservedRunningTime="2026-10-29 22:57:02.255635669 +0000 UTC m=+857.878546722"
Oct 29 22:57:02.676350 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.676320    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:57:02.679788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.679709    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 22:57:02.722839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.717083    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:57:02.728998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.728971    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 22:57:02.756468 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.756406    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podStartSLOduration=7.756386233 podStartE2EDuration="7.756386233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:02.299128036 +0000 UTC m=+857.922039059" watchObservedRunningTime="2026-10-29 22:57:02.756386233 +0000 UTC m=+858.379297253"
Oct 29 22:57:02.892398 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.892352    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:02.900565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:02.900521    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:02.904581 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:57:02.904522    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:57:03.011733 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:57:03.011694    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7304cb6_90c9_45c0_b6a4_5caec95733a6.slice/crio-conmon-b52ca8b7d95ce95c0118b453d0c5be467171cacccae9aa3f5b7b0f956b53854f.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:57:03.226413 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.226312    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" event={"ID":"61d297af-035a-421e-bf23-38d878371101","Type":"ContainerStarted","Data":"d814cbe1adbaffa35e08107787be07f3102568d36db71434c1e4af8967507da7"}
Oct 29 22:57:03.228181 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.228154    4827 generic.go:356] "Generic (PLEG): container finished" podID="a7304cb6-90c9-45c0-b6a4-5caec95733a6" containerID="b52ca8b7d95ce95c0118b453d0c5be467171cacccae9aa3f5b7b0f956b53854f" exitCode=0
Oct 29 22:57:03.228304 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.228237    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6" event={"ID":"a7304cb6-90c9-45c0-b6a4-5caec95733a6","Type":"ContainerDied","Data":"b52ca8b7d95ce95c0118b453d0c5be467171cacccae9aa3f5b7b0f956b53854f"}
Oct 29 22:57:03.230062 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.230038    4827 generic.go:356] "Generic (PLEG): container finished" podID="1dd433c4-f254-4087-a651-74ef9cc12d96" containerID="fb947ac7c49708d9166c7d556175d8119b4cf7789f6ddbc807309989a31c73e9" exitCode=0
Oct 29 22:57:03.230165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.230077    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5" event={"ID":"1dd433c4-f254-4087-a651-74ef9cc12d96","Type":"ContainerDied","Data":"fb947ac7c49708d9166c7d556175d8119b4cf7789f6ddbc807309989a31c73e9"}
Oct 29 22:57:03.232088 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.232068    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" event={"ID":"c197150f-64b3-41b9-a0ed-951e3cb3212d","Type":"ContainerStarted","Data":"4f62da5ddd3f9310faa1c140ef75a3f5e4e43704bfb3ddbcf922bef4882e1c5d"}
Oct 29 22:57:03.232683 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.232630    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:03.233695 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.233676    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:57:03.274320 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.274255    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podStartSLOduration=7.274235699 podStartE2EDuration="7.274235699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:03.271513676 +0000 UTC m=+858.894424697" watchObservedRunningTime="2026-10-29 22:57:03.274235699 +0000 UTC m=+858.897146720"
Oct 29 22:57:03.506082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.506004    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:03.523324 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.523250    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podStartSLOduration=6.523223908 podStartE2EDuration="6.523223908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:03.504199323 +0000 UTC m=+859.127110344" watchObservedRunningTime="2026-10-29 22:57:03.523223908 +0000 UTC m=+859.146134945"
Oct 29 22:57:03.585612 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:03.579221    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:04.027603 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.027565    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:04.029372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.029282    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:57:04.029372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.029330    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:57:04.244758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.243762    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:04.245267 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.245243    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:57:04.758810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.755422    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:04.760927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.760903    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:57:04.814865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.814781    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7304cb6-90c9-45c0-b6a4-5caec95733a6-kube-api-access-f67lp\" (UniqueName: \"kubernetes.io/projected/a7304cb6-90c9-45c0-b6a4-5caec95733a6-kube-api-access-f67lp\") pod \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") "
Oct 29 22:57:04.815021 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.814943    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd433c4-f254-4087-a651-74ef9cc12d96-kube-api-access-crs6w\" (UniqueName: \"kubernetes.io/projected/1dd433c4-f254-4087-a651-74ef9cc12d96-kube-api-access-crs6w\") pod \"1dd433c4-f254-4087-a651-74ef9cc12d96\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") "
Oct 29 22:57:04.815081 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.815033    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-xks-ca\") pod \"1dd433c4-f254-4087-a651-74ef9cc12d96\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") "
Oct 29 22:57:04.815144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.815086    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-openshift-ca\") pod \"1dd433c4-f254-4087-a651-74ef9cc12d96\" (UID: \"1dd433c4-f254-4087-a651-74ef9cc12d96\") "
Oct 29 22:57:04.815144 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.815119    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-openshift-ca\") pod \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") "
Oct 29 22:57:04.815245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.815179    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-xks-ca\") pod \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\" (UID: \"a7304cb6-90c9-45c0-b6a4-5caec95733a6\") "
Oct 29 22:57:04.817604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.817557    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-openshift-ca" pod "a7304cb6-90c9-45c0-b6a4-5caec95733a6" (UID: "a7304cb6-90c9-45c0-b6a4-5caec95733a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:04.819016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.818986    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-openshift-ca" pod "1dd433c4-f254-4087-a651-74ef9cc12d96" (UID: "1dd433c4-f254-4087-a651-74ef9cc12d96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:04.824182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.824153    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd433c4-f254-4087-a651-74ef9cc12d96-kube-api-access-crs6w" pod "1dd433c4-f254-4087-a651-74ef9cc12d96" (UID: "1dd433c4-f254-4087-a651-74ef9cc12d96"). InnerVolumeSpecName "kube-api-access-crs6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:04.824350 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.824328    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7304cb6-90c9-45c0-b6a4-5caec95733a6-kube-api-access-f67lp" pod "a7304cb6-90c9-45c0-b6a4-5caec95733a6" (UID: "a7304cb6-90c9-45c0-b6a4-5caec95733a6"). InnerVolumeSpecName "kube-api-access-f67lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:04.826972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.826917    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-xks-ca" pod "a7304cb6-90c9-45c0-b6a4-5caec95733a6" (UID: "a7304cb6-90c9-45c0-b6a4-5caec95733a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:04.829314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.829194    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-xks-ca" pod "1dd433c4-f254-4087-a651-74ef9cc12d96" (UID: "1dd433c4-f254-4087-a651-74ef9cc12d96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:04.848159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.848117    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"]
Oct 29 22:57:04.849712 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.849689    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a7304cb6-90c9-45c0-b6a4-5caec95733a6" containerName="revoke-keys"
Oct 29 22:57:04.849876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.849862    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7304cb6-90c9-45c0-b6a4-5caec95733a6" containerName="revoke-keys"
Oct 29 22:57:04.850000 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.849989    4827 cpu_manager.go:414] "containerMap: removing container" podUID="1dd433c4-f254-4087-a651-74ef9cc12d96" containerName="revoke-keys"
Oct 29 22:57:04.850078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.850067    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd433c4-f254-4087-a651-74ef9cc12d96" containerName="revoke-keys"
Oct 29 22:57:04.850378 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.850363    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7304cb6-90c9-45c0-b6a4-5caec95733a6" containerName="revoke-keys"
Oct 29 22:57:04.850484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.850472    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd433c4-f254-4087-a651-74ef9cc12d96" containerName="revoke-keys"
Oct 29 22:57:04.859816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.859254    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.884807 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.877674    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"]
Oct 29 22:57:04.959084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959048    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959258 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959109    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959258 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959153    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959318    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959424 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959386    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959680 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959656    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959800    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tk2\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-kube-api-access-t2tk2\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959940 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959925    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.959993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.959966    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8803d40a-f432-4846-a0e3-ae79296ec8d6-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:04.960216 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.960197    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:04.960269 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.960234    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67lp\" (UniqueName: \"kubernetes.io/projected/a7304cb6-90c9-45c0-b6a4-5caec95733a6-kube-api-access-f67lp\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:04.960382 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.960365    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crs6w\" (UniqueName: \"kubernetes.io/projected/1dd433c4-f254-4087-a651-74ef9cc12d96-kube-api-access-crs6w\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:04.960433 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.960390    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:04.960433 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.960405    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dd433c4-f254-4087-a651-74ef9cc12d96-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:04.960433 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:04.960425    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7304cb6-90c9-45c0-b6a4-5caec95733a6-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:05.028148 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.028118    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:57:05.028364 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.028342    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:57:05.061887 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.061852    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.061930    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tk2\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-kube-api-access-t2tk2\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062022    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062167 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062119    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8803d40a-f432-4846-a0e3-ae79296ec8d6-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062248 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062231    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062308 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062265    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062308 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062296    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062407 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062320    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062407 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062363    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062407 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062396    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062644 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062615    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.062707 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062676    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.063033 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.062934    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.063908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.063880    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8803d40a-f432-4846-a0e3-ae79296ec8d6-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.076708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.076281    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.100821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.079330    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.100821 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.093244    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.106613 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.106548    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tk2\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-kube-api-access-t2tk2\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.255321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.255287    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"
Oct 29 22:57:05.255321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.255297    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5" event={"ID":"1dd433c4-f254-4087-a651-74ef9cc12d96","Type":"ContainerDied","Data":"d6363bccfe6f27c650cd802e342b07475289a90f0bbb57e14e11b1d7244b3d1b"}
Oct 29 22:57:05.255939 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.255368    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6363bccfe6f27c650cd802e342b07475289a90f0bbb57e14e11b1d7244b3d1b"
Oct 29 22:57:05.256293 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.256273    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:05.257538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.257476    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6" event={"ID":"a7304cb6-90c9-45c0-b6a4-5caec95733a6","Type":"ContainerDied","Data":"04d13ec491d0d082694c3fce1742ac6a21714a27477886d05eb79d2d2a63b518"}
Oct 29 22:57:05.257645 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.257561    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d13ec491d0d082694c3fce1742ac6a21714a27477886d05eb79d2d2a63b518"
Oct 29 22:57:05.257929 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.257904    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"
Oct 29 22:57:05.451593 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.451545    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:05.451795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.451634    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:05.754832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.752683    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"]
Oct 29 22:57:05.802967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.801792    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:05.802967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.801888    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:05.802967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:05.801930    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:06.028235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:06.028147    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:57:06.028235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:06.028206    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:57:06.324874 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:06.323627    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" event={"ID":"8803d40a-f432-4846-a0e3-ae79296ec8d6","Type":"ContainerStarted","Data":"ffa002f5bf5e820908b63e5a1115cdceab0a8bb010b29686ae8f86c271de6bb5"}
Oct 29 22:57:06.324874 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:06.323699    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" event={"ID":"8803d40a-f432-4846-a0e3-ae79296ec8d6","Type":"ContainerStarted","Data":"5402b9ccaaf2b66dd7536108ad323d08e1aec06a41b839f7a79400b9df8a2eec"}
Oct 29 22:57:06.407565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:06.407500    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podStartSLOduration=2.4074357109999998 podStartE2EDuration="2.407435711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:06.401046663 +0000 UTC m=+862.023957684" watchObservedRunningTime="2026-10-29 22:57:06.407435711 +0000 UTC m=+862.030346732"
Oct 29 22:57:06.950797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:06.929911    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"]
Oct 29 22:57:07.214839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.213036    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.314801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.297210    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:07.365792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.353224    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:57:07.443789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.440501    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:57:07.443789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.440564    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:57:07.443789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.440571    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:07.443789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.440632    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:07.461874 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.449560    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:07.461874 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.449649    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"]
Oct 29 22:57:07.464464 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.464438    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"]
Oct 29 22:57:07.545438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.542558    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.546062 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.546032    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvgm\" (UniqueName: \"kubernetes.io/projected/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-kube-api-access-tpvgm\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.546225 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.546209    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad608fc-mt4g9"]
Oct 29 22:57:07.546589 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.546288    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.648000 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.647962    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvgm\" (UniqueName: \"kubernetes.io/projected/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-kube-api-access-tpvgm\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.648000 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.648013    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.648318 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.648065    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.650668 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.650642    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.654260 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.654235    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.721790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.711757    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvgm\" (UniqueName: \"kubernetes.io/projected/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-kube-api-access-tpvgm\") pod \"maas-api-revoke-sub-ade9c476-6b4sl\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:07.849866 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:07.849519    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:08.151925 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:08.132397    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:57:08.151925 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:08.132461    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:57:08.263789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:08.258188    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:08.263789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:08.258248    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:08.647991 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:08.646181    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99b86c5-f967-4a0a-8ea2-c3d839a7f914" path="/var/lib/kubelet/pods/e99b86c5-f967-4a0a-8ea2-c3d839a7f914/volumes"
Oct 29 22:57:08.730198 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:57:08.730160    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e439c78_2880_4ea8_9bfb_bdd0fe5efaeb.slice/crio-07c9a11f222b4bd9c702ed2607cac7ff0015e73b075f7f0e516aee261f27b545 WatchSource:0}: Error finding container 07c9a11f222b4bd9c702ed2607cac7ff0015e73b075f7f0e516aee261f27b545: Status 404 returned error can't find the container with id 07c9a11f222b4bd9c702ed2607cac7ff0015e73b075f7f0e516aee261f27b545
Oct 29 22:57:08.731845 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:08.731816    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"]
Oct 29 22:57:09.029091 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.028921    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:57:09.029091 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.028985    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:57:09.257535 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.257498    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:09.257689 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.257581    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:09.639628 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.639378    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl" event={"ID":"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb","Type":"ContainerStarted","Data":"09443086359dd15936c027e3c87a4e65a22e267c5ed3d7076b5e85d2f70b3216"}
Oct 29 22:57:09.639628 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.639429    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl" event={"ID":"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb","Type":"ContainerStarted","Data":"07c9a11f222b4bd9c702ed2607cac7ff0015e73b075f7f0e516aee261f27b545"}
Oct 29 22:57:09.710792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.710564    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl" podStartSLOduration=3.710541883 podStartE2EDuration="3.710541883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:09.710264214 +0000 UTC m=+865.333175259" watchObservedRunningTime="2026-10-29 22:57:09.710541883 +0000 UTC m=+865.333452940"
Oct 29 22:57:09.735422 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.735391    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"]
Oct 29 22:57:09.741230 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.741203    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:09.743356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.743326    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"]
Oct 29 22:57:09.791875 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.786919    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd525f\""
Oct 29 22:57:09.791875 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.787839    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd525f-dockercfg-tkxgt\""
Oct 29 22:57:09.974711 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.947642    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthwj\" (UniqueName: \"kubernetes.io/projected/93b04dd1-62ce-4858-b8e8-2ae0648b11af-kube-api-access-pthwj\") pod \"payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:09.974711 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:09.947866    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b04dd1-62ce-4858-b8e8-2ae0648b11af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:10.047833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.046738    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:57:10.047833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.046810    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:57:10.068160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.049438    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b04dd1-62ce-4858-b8e8-2ae0648b11af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:10.068160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.049658    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthwj\" (UniqueName: \"kubernetes.io/projected/93b04dd1-62ce-4858-b8e8-2ae0648b11af-kube-api-access-pthwj\") pod \"payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:10.068160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.066706    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b04dd1-62ce-4858-b8e8-2ae0648b11af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:10.159796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.158466    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthwj\" (UniqueName: \"kubernetes.io/projected/93b04dd1-62ce-4858-b8e8-2ae0648b11af-kube-api-access-pthwj\") pod \"payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:10.316307 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.290098    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:10.316307 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.290216    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:10.379480 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.378389    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:10.840803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.837713    4827 generic.go:356] "Generic (PLEG): container finished" podID="aaabce08-691b-4828-8810-c426708e784e" containerID="30e262dfc9ac5a7e450533f7af75562522ff03aa7d7d22892ddcf7705bfa3713" exitCode=0
Oct 29 22:57:10.840803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.837831    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" event={"ID":"aaabce08-691b-4828-8810-c426708e784e","Type":"ContainerDied","Data":"30e262dfc9ac5a7e450533f7af75562522ff03aa7d7d22892ddcf7705bfa3713"}
Oct 29 22:57:10.840803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.839942    4827 scope.go:122] "RemoveContainer" containerID="30e262dfc9ac5a7e450533f7af75562522ff03aa7d7d22892ddcf7705bfa3713"
Oct 29 22:57:10.842917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.842168    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:10.842917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.842236    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:10.842917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.842269    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:10.842917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.842667    4827 generic.go:356] "Generic (PLEG): container finished" podID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerID="2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3" exitCode=0
Oct 29 22:57:10.842917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.842740    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" event={"ID":"71fcbb11-1511-426f-9e8e-9052a7b55090","Type":"ContainerDied","Data":"2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3"}
Oct 29 22:57:10.843309 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.843270    4827 scope.go:122] "RemoveContainer" containerID="2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3"
Oct 29 22:57:10.844944 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.844901    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:10.845058 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.844986    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:10.845058 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:10.845035    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:11.033977 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.033493    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:11.034385 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.034369    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:11.035346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.035329    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:57:11.140930 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.140828    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"]
Oct 29 22:57:11.258852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.257985    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:11.258852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.258043    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:11.854829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.854798    4827 generic.go:356] "Generic (PLEG): container finished" podID="2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" containerID="09443086359dd15936c027e3c87a4e65a22e267c5ed3d7076b5e85d2f70b3216" exitCode=0
Oct 29 22:57:11.855312 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.854877    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl" event={"ID":"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb","Type":"ContainerDied","Data":"09443086359dd15936c027e3c87a4e65a22e267c5ed3d7076b5e85d2f70b3216"}
Oct 29 22:57:11.857128 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.857109    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" event={"ID":"aaabce08-691b-4828-8810-c426708e784e","Type":"ContainerStarted","Data":"2f8ca54a5e0481d05629c9b7710ef29ea315868b0c7afa0be1908e1b61a826d1"}
Oct 29 22:57:11.859378 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.859352    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" event={"ID":"71fcbb11-1511-426f-9e8e-9052a7b55090","Type":"ContainerStarted","Data":"e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0"}
Oct 29 22:57:11.860970 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.860950    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm" event={"ID":"93b04dd1-62ce-4858-b8e8-2ae0648b11af","Type":"ContainerStarted","Data":"2ee6ff975a6f21228d1511b837e7cda956fabbfea807b1a0c79d329343049550"}
Oct 29 22:57:11.861090 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.861007    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm" event={"ID":"93b04dd1-62ce-4858-b8e8-2ae0648b11af","Type":"ContainerStarted","Data":"528a87e0b81c764737c69eb3b2df213ad44e9f6d7bd4ec4e303fd7a4243ccfc7"}
Oct 29 22:57:11.861162 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:11.861145    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:12.050408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.049114    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:57:12.057080 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.050926    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:12.057080 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.050975    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:12.123082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.117929    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm" podStartSLOduration=3.117907369 podStartE2EDuration="3.117907369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:12.070583848 +0000 UTC m=+867.693494903" watchObservedRunningTime="2026-10-29 22:57:12.117907369 +0000 UTC m=+867.740818390"
Oct 29 22:57:12.331798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.260476    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:12.331798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.260540    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:12.767726 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.767691    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:57:12.768135 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.768112    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:12.768349 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:12.768324    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:13.023850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.013758    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:13.023850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.013833    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:13.257886 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.257834    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:13.258067 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.257897    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:13.653880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.653851    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:13.658834 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.658810    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-xks-ca\") pod \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") "
Oct 29 22:57:13.658998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.658987    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-kube-api-access-tpvgm\" (UniqueName: \"kubernetes.io/projected/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-kube-api-access-tpvgm\") pod \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") "
Oct 29 22:57:13.659277 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.659257    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-openshift-ca\") pod \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\" (UID: \"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb\") "
Oct 29 22:57:13.659566 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.659544    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-openshift-ca" pod "2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" (UID: "2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:13.660138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.660119    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:13.661027 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.660977    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-xks-ca" pod "2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" (UID: "2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:13.661262 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.661241    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-kube-api-access-tpvgm" pod "2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" (UID: "2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb"). InnerVolumeSpecName "kube-api-access-tpvgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:13.760098 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.760012    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:13.760098 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.760075    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:13.790437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.790402    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:13.790437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:13.790437    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvgm\" (UniqueName: \"kubernetes.io/projected/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb-kube-api-access-tpvgm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:14.024890 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.017548    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:14.024890 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.017604    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:14.068604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.067391    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl" event={"ID":"2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb","Type":"ContainerDied","Data":"07c9a11f222b4bd9c702ed2607cac7ff0015e73b075f7f0e516aee261f27b545"}
Oct 29 22:57:14.068604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.067458    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c9a11f222b4bd9c702ed2607cac7ff0015e73b075f7f0e516aee261f27b545"
Oct 29 22:57:14.068604 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.067533    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"
Oct 29 22:57:14.258393 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.258359    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:57:14.258664 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.258624    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:57:14.757814 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.757759    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:14.758001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:14.757844    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:15.006209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:15.006172    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:15.006490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:15.006460    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:15.758512 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:15.758475    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:15.759046 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:15.758535    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:15.847461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:15.847405    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:15.848064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:15.848036    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:15.848957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:15.848936    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 22:57:16.026576 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:16.026440    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:16.026918 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:16.026849    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:16.212996 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:16.212961    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:57:16.757889 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:16.757740    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:16.757889 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:16.757807    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:17.006825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:17.006789    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:17.008244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:17.008170    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:17.757754 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:17.757657    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:17.757754 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:17.757714    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:18.005241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.005197    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:18.005437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.005263    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:18.067481 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.067393    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:18.184791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.174922    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:18.289321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.289263    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:18.518787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.515685    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:18.762314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.762280    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:18.762605 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.762582    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:18.929173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.929132    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 29 22:57:18.948976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:18.948934    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:19.016795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.015215    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:19.016795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.015294    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:19.036956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.036912    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:57:19.098411 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.098371    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:57:19.121871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.121833    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"]
Oct 29 22:57:19.204879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.184321    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:19.204879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.184886    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:57:19.212794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.210129    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:57:19.320791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.320234    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:57:19.378791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.367937    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:57:19.393740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.393696    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:19.406020 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.405986    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:57:19.464905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.436911    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:19.505165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.505132    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"]
Oct 29 22:57:19.729849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.645912    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:19.729849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.647673    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:57:19.773020 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.744554    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:19.918414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.918357    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:19.919979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.919953    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:19.920219 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.920186    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:19.951928 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:19.951888    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.000935 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.000829    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:57:20.027655 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.011879    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:20.110160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.084493    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.112257 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.112204    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:20.112448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.110005    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.143055 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.143014    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:57:20.144518 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.144498    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:57:20.173835 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.163091    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:57:20.174097 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.109919    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.195311 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.195280    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.253864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.250518    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.253864 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.251059    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:57:20.340792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.324454    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.422668 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.422619    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:20.447259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.447229    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:57:20.503120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.503065    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:57:20.529842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.504460    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:57:20.800849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.800069    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:20.928793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.923068    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:20.942794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.936861    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:57:20.968793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.944924    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:20.968793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:20.944976    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:21.124993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.065590    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c874f" podUID="89beb379-c7e3-441a-90cb-5572b7d934da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.24:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:21.124993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.102978    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:21.152897 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.150250    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:57:21.221792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.205890    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:57:21.246895 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.246847    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 22:57:21.410439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.289980    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:21.410439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.291348    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:57:21.410439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.301291    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:21.410439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.305538    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x" podUID="f90fd47c-4d11-4faa-8eb9-d394960cc2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:21.410439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.353387    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e815205-dl6lx"]
Oct 29 22:57:21.410439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.367886    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:21.410439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.399357    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:57:21.571829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.508670    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:21.571829 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.562890    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:21.810455 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.402007    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:57:21.810455 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.800019    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:57:21.810455 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.800113    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580d6ee8-744rd"]
Oct 29 22:57:21.952791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.901714    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:21.999801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:21.956544    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:22.153879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.124947    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:22.179051 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.160876    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:22.332889 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.331815    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:22.423852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.355629    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:22.423852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.356129    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:57:22.507793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.442285    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:22.655306 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.508801    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:22.655596 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.655550    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 22:57:22.658857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.525900    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:22.659034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.525862    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:22.659326 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:22.570786    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.081822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.076442    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:23.081822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.076680    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:23.137919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.137870    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.167898 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.167854    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.182591 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.181242    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.340120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.323152    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa11da47-5ad8-4944-99bf-f20bd8cadd10" path="/var/lib/kubelet/pods/aa11da47-5ad8-4944-99bf-f20bd8cadd10/volumes"
Oct 29 22:57:23.377810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.355907    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.381627 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.381591    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:57:23.397234 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.397180    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:57:23.408739 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.349859    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:57:23.418832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.412885    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:23.419116 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.419086    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.420448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.419297    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:23.420617 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.420594    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.454725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.454687    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac43fdf9-f124-40c2-abc0-7f08ee6c60d7" path="/var/lib/kubelet/pods/ac43fdf9-f124-40c2-abc0-7f08ee6c60d7/volumes"
Oct 29 22:57:23.455487 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.455464    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:57:23.501250 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.501213    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:23.501516 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.501486    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.660075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.660030    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:57:23.665802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.661813    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.701337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.701300    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.823946 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.823911    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:23.857361 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.843866    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:23.928725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.928625    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 29 22:57:23.929710 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.929687    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:57:23.940286 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.939895    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:23.940286 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:23.939949    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.091895 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.086086    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.091895 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.086621    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.179848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.176579    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:24.205791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.181964    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.401792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.376720    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.402236 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.402170    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.426791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.403313    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:57:24.427081 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.407840    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:24.427268 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.427232    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.452850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.414778    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:57:24.535901 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.535862    4827 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:24.536165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.536137    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.801142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.801042    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:57:24.841808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.813297    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:24.841808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.816313    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:57:24.955825 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.955762    4827 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:57:24.956133 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:24.956087    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:57:25.098065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.097958    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:25.098344 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.098316    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:25.233793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.228881    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:25.233793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.229278    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:57:25.295322 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.295284    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:57:25.297183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.297152    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:25.364417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.364332    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:57:25.421824 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.406856    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:25.506244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.406918    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:57:25.506244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.433211    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:57:25.506244 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.469345    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:25.516113 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.516082    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 22:57:25.819957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.819904    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:25.894522 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.889409    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:57:25.907505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:25.901658    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:26.063089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.060061    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:57:26.063089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.060944    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:57:26.132655 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.117858    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:26.132655 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.127238    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:26.149221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.148294    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:57:26.171876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.171848    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:57:26.242060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.238923    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:57:26.242060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.238976    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:57:26.242060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.241498    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:57:26.258611 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.254985    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:57:26.261818 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.260941    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:57:26.262116 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:26.262096    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:57:29.520939 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.520903    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"]
Oct 29 22:57:29.521967 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.521951    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" containerName="revoke-keys"
Oct 29 22:57:29.522017 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.521970    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" containerName="revoke-keys"
Oct 29 22:57:29.522126 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.522115    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" containerName="revoke-keys"
Oct 29 22:57:29.527783 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.527747    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.542234 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.542206    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"]
Oct 29 22:57:29.547355 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.547328    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:29.547483 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.547466    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:57:29.609457 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.609425    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.609625 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.609551    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt6l\" (UniqueName: \"kubernetes.io/projected/9f6af75e-c9de-4854-9b4f-89370453d05e-kube-api-access-fvt6l\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.609625 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.609619    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.714906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.714872    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.715235 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.715217    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt6l\" (UniqueName: \"kubernetes.io/projected/9f6af75e-c9de-4854-9b4f-89370453d05e-kube-api-access-fvt6l\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.715436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.715419    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.725965 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.722805    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.771378 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.769598    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.775097 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.775064    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt6l\" (UniqueName: \"kubernetes.io/projected/9f6af75e-c9de-4854-9b4f-89370453d05e-kube-api-access-fvt6l\") pod \"maas-api-revoke-sub-fbe08053-gnw64\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:29.844281 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:29.844253    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:30.046424 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:30.046341    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"]
Oct 29 22:57:30.051402 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:30.051376    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5879ada-69qm7"]
Oct 29 22:57:30.162400 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:57:30.162356    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6af75e_c9de_4854_9b4f_89370453d05e.slice/crio-d48a1be36d458673e4434b73c019aaf32ee55ebb4531f268d6511df0ab3c971c WatchSource:0}: Error finding container d48a1be36d458673e4434b73c019aaf32ee55ebb4531f268d6511df0ab3c971c: Status 404 returned error can't find the container with id d48a1be36d458673e4434b73c019aaf32ee55ebb4531f268d6511df0ab3c971c
Oct 29 22:57:30.164136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:30.164104    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"]
Oct 29 22:57:30.302712 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:30.302008    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64" event={"ID":"9f6af75e-c9de-4854-9b4f-89370453d05e","Type":"ContainerStarted","Data":"d48a1be36d458673e4434b73c019aaf32ee55ebb4531f268d6511df0ab3c971c"}
Oct 29 22:57:30.492415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:30.492376    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f682e56-8ac0-4383-b884-2a84f47c07d2" path="/var/lib/kubelet/pods/8f682e56-8ac0-4383-b884-2a84f47c07d2/volumes"
Oct 29 22:57:31.315975 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:31.315943    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64" event={"ID":"9f6af75e-c9de-4854-9b4f-89370453d05e","Type":"ContainerStarted","Data":"d968acade4df5849b85b5c6a07b26aacc05e67d1a20cf38f5a925244046dfbd8"}
Oct 29 22:57:31.357565 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:31.357506    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64" podStartSLOduration=2.35748851 podStartE2EDuration="2.35748851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:31.356378693 +0000 UTC m=+886.979289713" watchObservedRunningTime="2026-10-29 22:57:31.35748851 +0000 UTC m=+886.980399531"
Oct 29 22:57:32.331786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.331190    4827 generic.go:356] "Generic (PLEG): container finished" podID="9f6af75e-c9de-4854-9b4f-89370453d05e" containerID="d968acade4df5849b85b5c6a07b26aacc05e67d1a20cf38f5a925244046dfbd8" exitCode=0
Oct 29 22:57:32.331786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.331323    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64" event={"ID":"9f6af75e-c9de-4854-9b4f-89370453d05e","Type":"ContainerDied","Data":"d968acade4df5849b85b5c6a07b26aacc05e67d1a20cf38f5a925244046dfbd8"}
Oct 29 22:57:32.729239 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.729192    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"]
Oct 29 22:57:32.815539 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.815507    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.818564 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.818534    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"]
Oct 29 22:57:32.887590 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:57:32.887545    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:57:32.887811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.887680    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pc2x\" (UniqueName: \"kubernetes.io/projected/acfd6131-e8ec-46af-88e2-c20e52e21c1b-kube-api-access-6pc2x\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.887811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.887797    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.888021 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.888001    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.988879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.988788    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pc2x\" (UniqueName: \"kubernetes.io/projected/acfd6131-e8ec-46af-88e2-c20e52e21c1b-kube-api-access-6pc2x\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.988879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.988856    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.989121 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.988960    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.989750 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.989718    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:32.991290 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:32.991269    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:33.012261 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.012228    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pc2x\" (UniqueName: \"kubernetes.io/projected/acfd6131-e8ec-46af-88e2-c20e52e21c1b-kube-api-access-6pc2x\") pod \"maas-api-revoke-sub-771d0b93-g5wp7\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:33.188543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.188500    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:33.533294 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.529352    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"]
Oct 29 22:57:33.543019 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.542999    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:33.617791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.611321    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-openshift-ca" pod "9f6af75e-c9de-4854-9b4f-89370453d05e" (UID: "9f6af75e-c9de-4854-9b4f-89370453d05e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:33.617791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.611688    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-openshift-ca\") pod \"9f6af75e-c9de-4854-9b4f-89370453d05e\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") "
Oct 29 22:57:33.617791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.612396    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6af75e-c9de-4854-9b4f-89370453d05e-kube-api-access-fvt6l\" (UniqueName: \"kubernetes.io/projected/9f6af75e-c9de-4854-9b4f-89370453d05e-kube-api-access-fvt6l\") pod \"9f6af75e-c9de-4854-9b4f-89370453d05e\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") "
Oct 29 22:57:33.617791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.613461    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-xks-ca\") pod \"9f6af75e-c9de-4854-9b4f-89370453d05e\" (UID: \"9f6af75e-c9de-4854-9b4f-89370453d05e\") "
Oct 29 22:57:33.617791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.614457    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:33.702184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.667051    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-xks-ca" pod "9f6af75e-c9de-4854-9b4f-89370453d05e" (UID: "9f6af75e-c9de-4854-9b4f-89370453d05e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:33.702184 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.689203    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6af75e-c9de-4854-9b4f-89370453d05e-kube-api-access-fvt6l" pod "9f6af75e-c9de-4854-9b4f-89370453d05e" (UID: "9f6af75e-c9de-4854-9b4f-89370453d05e"). InnerVolumeSpecName "kube-api-access-fvt6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:33.738440 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.716011    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6af75e-c9de-4854-9b4f-89370453d05e-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:33.738440 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:33.716048    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvt6l\" (UniqueName: \"kubernetes.io/projected/9f6af75e-c9de-4854-9b4f-89370453d05e-kube-api-access-fvt6l\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:34.351104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:34.351019    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7" event={"ID":"acfd6131-e8ec-46af-88e2-c20e52e21c1b","Type":"ContainerStarted","Data":"b809d103950e410acc1464a4c6391eade1d23622cc359b8c9a4772949bf563d6"}
Oct 29 22:57:34.351104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:34.351076    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7" event={"ID":"acfd6131-e8ec-46af-88e2-c20e52e21c1b","Type":"ContainerStarted","Data":"f13196a899894b7f81a93fe7518bc5e5351148d281fa66bec98504aaf98c2524"}
Oct 29 22:57:34.352744 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:34.352720    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"
Oct 29 22:57:34.352919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:34.352749    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64" event={"ID":"9f6af75e-c9de-4854-9b4f-89370453d05e","Type":"ContainerDied","Data":"d48a1be36d458673e4434b73c019aaf32ee55ebb4531f268d6511df0ab3c971c"}
Oct 29 22:57:34.352919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:34.352801    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d48a1be36d458673e4434b73c019aaf32ee55ebb4531f268d6511df0ab3c971c"
Oct 29 22:57:34.395467 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:34.395403    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7" podStartSLOduration=2.395386031 podStartE2EDuration="2.395386031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:34.390071774 +0000 UTC m=+890.012982796" watchObservedRunningTime="2026-10-29 22:57:34.395386031 +0000 UTC m=+890.018297054"
Oct 29 22:57:35.448960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.448919    4827 generic.go:356] "Generic (PLEG): container finished" podID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerID="f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3" exitCode=0
Oct 29 22:57:35.449839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.448986    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" event={"ID":"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b","Type":"ContainerDied","Data":"f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3"}
Oct 29 22:57:35.451758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.451740    4827 scope.go:122] "RemoveContainer" containerID="f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3"
Oct 29 22:57:35.454729 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.454691    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:35.455645 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.455217    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:35.455645 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.455280    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:35.456870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.456748    4827 generic.go:356] "Generic (PLEG): container finished" podID="acfd6131-e8ec-46af-88e2-c20e52e21c1b" containerID="b809d103950e410acc1464a4c6391eade1d23622cc359b8c9a4772949bf563d6" exitCode=0
Oct 29 22:57:35.456870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:35.456794    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7" event={"ID":"acfd6131-e8ec-46af-88e2-c20e52e21c1b","Type":"ContainerDied","Data":"b809d103950e410acc1464a4c6391eade1d23622cc359b8c9a4772949bf563d6"}
Oct 29 22:57:36.394129 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.394099    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"]
Oct 29 22:57:36.394815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.394800    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9f6af75e-c9de-4854-9b4f-89370453d05e" containerName="revoke-keys"
Oct 29 22:57:36.394863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.394816    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6af75e-c9de-4854-9b4f-89370453d05e" containerName="revoke-keys"
Oct 29 22:57:36.394966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.394956    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6af75e-c9de-4854-9b4f-89370453d05e" containerName="revoke-keys"
Oct 29 22:57:36.399908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.399889    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:36.414948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.414598    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a0dbf5-dockercfg-s9gxp\""
Oct 29 22:57:36.414948 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.414735    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a0dbf5\""
Oct 29 22:57:36.416725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.416702    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"]
Oct 29 22:57:36.499157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.499123    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285da0f6-1f6d-4541-a5d7-d09d2297a793-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:36.499886 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.499860    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxlb\" (UniqueName: \"kubernetes.io/projected/285da0f6-1f6d-4541-a5d7-d09d2297a793-kube-api-access-bsxlb\") pod \"payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:36.509753 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.509733    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"]
Oct 29 22:57:36.523803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.517856    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" event={"ID":"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b","Type":"ContainerStarted","Data":"51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9"}
Oct 29 22:57:36.523803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.518131    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"]
Oct 29 22:57:36.523803 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.518235    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:36.651645 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.603980    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285da0f6-1f6d-4541-a5d7-d09d2297a793-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:36.693805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.661097    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6spm\" (UniqueName: \"kubernetes.io/projected/8c182a50-c02c-4cb9-ada5-181183fd0118-kube-api-access-c6spm\") pod \"payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") " pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:36.694127 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.671278    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:57:36.694375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.609618    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285da0f6-1f6d-4541-a5d7-d09d2297a793-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:36.694547 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.694521    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxlb\" (UniqueName: \"kubernetes.io/projected/285da0f6-1f6d-4541-a5d7-d09d2297a793-kube-api-access-bsxlb\") pod \"payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:36.695019 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.694982    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c182a50-c02c-4cb9-ada5-181183fd0118-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") " pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:36.808793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.808262    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c182a50-c02c-4cb9-ada5-181183fd0118-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") " pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:36.808793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.808603    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6spm\" (UniqueName: \"kubernetes.io/projected/8c182a50-c02c-4cb9-ada5-181183fd0118-kube-api-access-c6spm\") pod \"payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") " pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:36.811850 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.810330    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c182a50-c02c-4cb9-ada5-181183fd0118-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") " pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:36.899088 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.896020    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6spm\" (UniqueName: \"kubernetes.io/projected/8c182a50-c02c-4cb9-ada5-181183fd0118-kube-api-access-c6spm\") pod \"payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") " pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:36.960262 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:36.954107    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxlb\" (UniqueName: \"kubernetes.io/projected/285da0f6-1f6d-4541-a5d7-d09d2297a793-kube-api-access-bsxlb\") pod \"payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:37.141229 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.141188    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:57:37.155876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.147415    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:37.181102 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.160687    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:37.181102 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.180998    4827 patch_prober.go:29] interesting pod/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.181102 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.181046    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:57:37.354124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.354095    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:37.449717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.448963    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfd6131-e8ec-46af-88e2-c20e52e21c1b-kube-api-access-6pc2x\" (UniqueName: \"kubernetes.io/projected/acfd6131-e8ec-46af-88e2-c20e52e21c1b-kube-api-access-6pc2x\") pod \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") "
Oct 29 22:57:37.449717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.449040    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-xks-ca\") pod \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") "
Oct 29 22:57:37.449717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.449082    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-openshift-ca\") pod \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\" (UID: \"acfd6131-e8ec-46af-88e2-c20e52e21c1b\") "
Oct 29 22:57:37.449717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.449574    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-openshift-ca" pod "acfd6131-e8ec-46af-88e2-c20e52e21c1b" (UID: "acfd6131-e8ec-46af-88e2-c20e52e21c1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:37.449978 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.449827    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:37.451240 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.451208    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-xks-ca" pod "acfd6131-e8ec-46af-88e2-c20e52e21c1b" (UID: "acfd6131-e8ec-46af-88e2-c20e52e21c1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:37.451345 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.451278    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfd6131-e8ec-46af-88e2-c20e52e21c1b-kube-api-access-6pc2x" pod "acfd6131-e8ec-46af-88e2-c20e52e21c1b" (UID: "acfd6131-e8ec-46af-88e2-c20e52e21c1b"). InnerVolumeSpecName "kube-api-access-6pc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:37.578847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.578183    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pc2x\" (UniqueName: \"kubernetes.io/projected/acfd6131-e8ec-46af-88e2-c20e52e21c1b-kube-api-access-6pc2x\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:37.578847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.578216    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd6131-e8ec-46af-88e2-c20e52e21c1b-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:57:37.740490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.738631    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"
Oct 29 22:57:37.740490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.738691    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7" event={"ID":"acfd6131-e8ec-46af-88e2-c20e52e21c1b","Type":"ContainerDied","Data":"f13196a899894b7f81a93fe7518bc5e5351148d281fa66bec98504aaf98c2524"}
Oct 29 22:57:37.740490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.738745    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13196a899894b7f81a93fe7518bc5e5351148d281fa66bec98504aaf98c2524"
Oct 29 22:57:37.740490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.740419    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"]
Oct 29 22:57:37.742564 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:57:37.742539    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c182a50_c02c_4cb9_ada5_181183fd0118.slice/crio-116b8b30fcbe5ce9368eabc953fa227f79ec1a081d4aebd584d307e28e31c421 WatchSource:0}: Error finding container 116b8b30fcbe5ce9368eabc953fa227f79ec1a081d4aebd584d307e28e31c421: Status 404 returned error can't find the container with id 116b8b30fcbe5ce9368eabc953fa227f79ec1a081d4aebd584d307e28e31c421
Oct 29 22:57:37.747143 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:37.747126    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"]
Oct 29 22:57:38.090306 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.090223    4827 patch_prober.go:29] interesting pod/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:38.090306 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.090270    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:57:38.749143 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.749104    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7" event={"ID":"285da0f6-1f6d-4541-a5d7-d09d2297a793","Type":"ContainerStarted","Data":"9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8"}
Oct 29 22:57:38.749633 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.749222    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:38.749633 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.749255    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7" event={"ID":"285da0f6-1f6d-4541-a5d7-d09d2297a793","Type":"ContainerStarted","Data":"98aa7a055491e9c79790764f2a4e38e5291d0efcf3457c79a1cd724fd5aae1c2"}
Oct 29 22:57:38.750941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.750909    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx" event={"ID":"8c182a50-c02c-4cb9-ada5-181183fd0118","Type":"ContainerStarted","Data":"11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704"}
Oct 29 22:57:38.751079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.750959    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx" event={"ID":"8c182a50-c02c-4cb9-ada5-181183fd0118","Type":"ContainerStarted","Data":"116b8b30fcbe5ce9368eabc953fa227f79ec1a081d4aebd584d307e28e31c421"}
Oct 29 22:57:38.751079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.750980    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:38.810034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.809966    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7" podStartSLOduration=2.809949372 podStartE2EDuration="2.809949372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:38.808371542 +0000 UTC m=+894.431282562" watchObservedRunningTime="2026-10-29 22:57:38.809949372 +0000 UTC m=+894.432860393"
Oct 29 22:57:38.853714 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:38.853665    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx" podStartSLOduration=2.853652948 podStartE2EDuration="2.853652948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:38.850036948 +0000 UTC m=+894.472947970" watchObservedRunningTime="2026-10-29 22:57:38.853652948 +0000 UTC m=+894.476563969"
Oct 29 22:57:39.090246 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:39.090159    4827 patch_prober.go:29] interesting pod/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:39.090489 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:39.090450    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:57:40.090683 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:40.090647    4827 patch_prober.go:29] interesting pod/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:40.091285 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:40.090711    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:57:41.122048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:41.104106    4827 patch_prober.go:29] interesting pod/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:41.122048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:41.113665    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:57:42.119942 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:42.119909    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:57:42.122606 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:42.122580    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:57:44.619797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:44.619759    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:57:44.625347 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:44.625322    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 22:57:44.998165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:44.998135    4827 scope.go:122] "RemoveContainer" containerID="8bc2319d713ab28d969040c67e764544ff35f0518bf9b95a26678ce88316d5a5"
Oct 29 22:57:45.009417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.009392    4827 scope.go:122] "RemoveContainer" containerID="16617e257da7ac10a1e67c682d564fd1c16db19a363bf389f959b0b5294ef3d0"
Oct 29 22:57:45.019191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.019171    4827 scope.go:122] "RemoveContainer" containerID="3c98a22a12aa71c7ccc9720fa411f5ef4abf8b4795802033a110eac86ba88e85"
Oct 29 22:57:45.028743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.028724    4827 scope.go:122] "RemoveContainer" containerID="6cad1252710efef7973e047b9f0269245afb03713aaab88082d42730eda4352e"
Oct 29 22:57:45.244386 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.244355    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"]
Oct 29 22:57:45.245147 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.245125    4827 cpu_manager.go:414] "containerMap: removing container" podUID="acfd6131-e8ec-46af-88e2-c20e52e21c1b" containerName="revoke-keys"
Oct 29 22:57:45.245147 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.245144    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfd6131-e8ec-46af-88e2-c20e52e21c1b" containerName="revoke-keys"
Oct 29 22:57:45.245370 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.245351    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfd6131-e8ec-46af-88e2-c20e52e21c1b" containerName="revoke-keys"
Oct 29 22:57:45.251570 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.251506    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.269677 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.269654    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-36befd-openshift-default-dockercfg-zwt8s\""
Oct 29 22:57:45.273359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.273335    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"]
Oct 29 22:57:45.400041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400010    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-credential-socket\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400072    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-envoy\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400135    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b985efd-ab05-4960-ae31-13c61bf74b1b-istiod-ca-cert\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400181    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmbp\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-kube-api-access-blmbp\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400336 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400317    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-certs\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400463 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400448    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-podinfo\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400485    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-socket\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400537 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400504    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-token\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.400570 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.400542    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-data\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509036    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-certs\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509191    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-podinfo\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509239    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-socket\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509267    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-token\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509301    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-data\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509382    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-credential-socket\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509477    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-envoy\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509503    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b985efd-ab05-4960-ae31-13c61bf74b1b-istiod-ca-cert\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.509839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.509566    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmbp\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-kube-api-access-blmbp\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.510446 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.510381    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-certs\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.525840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.525787    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-credential-socket\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.526614 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.526585    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-socket\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.527350 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.527328    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-data\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.528493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.528468    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b985efd-ab05-4960-ae31-13c61bf74b1b-istiod-ca-cert\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.549804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.547135    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-envoy\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.570792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.570572    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-podinfo\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.584789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.575446    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-token\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.584789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.576221    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmbp\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-kube-api-access-blmbp\") pod \"e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") " pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:45.882815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:45.877556    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:46.293307 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:57:46.293262    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b985efd_ab05_4960_ae31_13c61bf74b1b.slice/crio-cee1db23b0d2b1e843d3dc6a0f0490894b664f2cb059b21cd8e19a168fa8788e WatchSource:0}: Error finding container cee1db23b0d2b1e843d3dc6a0f0490894b664f2cb059b21cd8e19a168fa8788e: Status 404 returned error can't find the container with id cee1db23b0d2b1e843d3dc6a0f0490894b664f2cb059b21cd8e19a168fa8788e
Oct 29 22:57:46.294084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.294057    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"]
Oct 29 22:57:46.296251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.296212    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:46.296365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.296294    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:46.296365 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.296323    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:46.863658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.863616    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" event={"ID":"8b985efd-ab05-4960-ae31-13c61bf74b1b","Type":"ContainerStarted","Data":"475fd2bdd94f3d24ba1d52b3c50bb0312f38b15ad30df1884b58b42090e204e8"}
Oct 29 22:57:46.863896 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.863671    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" event={"ID":"8b985efd-ab05-4960-ae31-13c61bf74b1b","Type":"ContainerStarted","Data":"cee1db23b0d2b1e843d3dc6a0f0490894b664f2cb059b21cd8e19a168fa8788e"}
Oct 29 22:57:46.877952 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.877892    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:46.921185 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:46.921127    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podStartSLOduration=1.921106227 podStartE2EDuration="1.921106227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:46.919531328 +0000 UTC m=+902.542442349" watchObservedRunningTime="2026-10-29 22:57:46.921106227 +0000 UTC m=+902.544017248"
Oct 29 22:57:47.764907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.764754    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"]
Oct 29 22:57:47.771835 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.771809    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.787554 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.787525    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a1d1da-openshift-default-dockercfg-vb57s\""
Oct 29 22:57:47.806397 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.806366    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"]
Oct 29 22:57:47.923193 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.923155    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.940414    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-token\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.940611    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-podinfo\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.940689    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-data\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.940721    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-envoy\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.940961    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-socket\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.940991    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-certs\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.941027    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjlvg\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-kube-api-access-kjlvg\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.941067    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-credential-socket\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.958790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.941178    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79b06ae-bc79-45a0-acec-9462a314fba1-istiod-ca-cert\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:47.960241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.960215    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:47.990856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:47.987036    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.100969    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-data\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.085759    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-data\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.101394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-envoy\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.101644    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-socket\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.101683    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-certs\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.101740    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjlvg\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-kube-api-access-kjlvg\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.101798    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-credential-socket\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.101968    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79b06ae-bc79-45a0-acec-9462a314fba1-istiod-ca-cert\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.102069    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-token\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.102406    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-podinfo\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.104075    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-socket\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.107837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.104380    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-certs\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.125705 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.113463    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79b06ae-bc79-45a0-acec-9462a314fba1-istiod-ca-cert\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.127232 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.127201    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-credential-socket\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.134499 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.134327    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-envoy\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.536820 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.535465    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-podinfo\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.677907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.677877    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjlvg\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-kube-api-access-kjlvg\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.685558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.685525    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-token\") pod \"e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") " pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:48.697078 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:48.697043    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:49.021935 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:49.021906    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"]
Oct 29 22:57:49.022383 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:57:49.021912    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79b06ae_bc79_45a0_acec_9462a314fba1.slice/crio-6f92bce815da7a94ad622961672a4fb7c218becbcbb67894fe0a9f7e9ef8cd48 WatchSource:0}: Error finding container 6f92bce815da7a94ad622961672a4fb7c218becbcbb67894fe0a9f7e9ef8cd48: Status 404 returned error can't find the container with id 6f92bce815da7a94ad622961672a4fb7c218becbcbb67894fe0a9f7e9ef8cd48
Oct 29 22:57:49.025066 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:49.025027    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:49.025157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:49.025098    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:49.025157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:49.025131    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:57:49.786787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:49.786032    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:57:49.803789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:49.800205    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:57:50.140457 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:50.140415    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" event={"ID":"a79b06ae-bc79-45a0-acec-9462a314fba1","Type":"ContainerStarted","Data":"b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44"}
Oct 29 22:57:50.140872 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:50.140483    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" event={"ID":"a79b06ae-bc79-45a0-acec-9462a314fba1","Type":"ContainerStarted","Data":"6f92bce815da7a94ad622961672a4fb7c218becbcbb67894fe0a9f7e9ef8cd48"}
Oct 29 22:57:50.188482 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:50.188428    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podStartSLOduration=3.18841419 podStartE2EDuration="3.18841419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:50.184437734 +0000 UTC m=+905.807348754" watchObservedRunningTime="2026-10-29 22:57:50.18841419 +0000 UTC m=+905.811325209"
Oct 29 22:57:50.697531 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:50.697499    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:50.702801 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:50.702779    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:51.148629 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.148601    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:51.149561 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.149539    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:57:51.540844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.540283    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"]
Oct 29 22:57:51.584822 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.577475    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:51.621529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.618626    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"]
Oct 29 22:57:51.650788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.648184    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36befd\""
Oct 29 22:57:51.650788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.648387    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-36befd-dockercfg-slzrj\""
Oct 29 22:57:51.783496 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.782098    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe98308-7569-4c57-80c2-83df2ee955e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:51.783496 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.782223    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhwrd\" (UniqueName: \"kubernetes.io/projected/dbe98308-7569-4c57-80c2-83df2ee955e6-kube-api-access-fhwrd\") pod \"payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:51.904529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.884433    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe98308-7569-4c57-80c2-83df2ee955e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:51.904529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.884586    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhwrd\" (UniqueName: \"kubernetes.io/projected/dbe98308-7569-4c57-80c2-83df2ee955e6-kube-api-access-fhwrd\") pod \"payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:51.904529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.885996    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe98308-7569-4c57-80c2-83df2ee955e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:51.939788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:51.934833    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhwrd\" (UniqueName: \"kubernetes.io/projected/dbe98308-7569-4c57-80c2-83df2ee955e6-kube-api-access-fhwrd\") pod \"payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:52.079802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:52.079616    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:52.506482 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:57:52.506452    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe98308_7569_4c57_80c2_83df2ee955e6.slice/crio-8fd0d84a202e5bd1e99918356257dc6be857bf99c0fb637cd86defe8c93a0c4e WatchSource:0}: Error finding container 8fd0d84a202e5bd1e99918356257dc6be857bf99c0fb637cd86defe8c93a0c4e: Status 404 returned error can't find the container with id 8fd0d84a202e5bd1e99918356257dc6be857bf99c0fb637cd86defe8c93a0c4e
Oct 29 22:57:52.506934 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:52.506864    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"]
Oct 29 22:57:53.201993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:53.201946    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g" event={"ID":"dbe98308-7569-4c57-80c2-83df2ee955e6","Type":"ContainerStarted","Data":"ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480"}
Oct 29 22:57:53.202189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:53.202010    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g" event={"ID":"dbe98308-7569-4c57-80c2-83df2ee955e6","Type":"ContainerStarted","Data":"8fd0d84a202e5bd1e99918356257dc6be857bf99c0fb637cd86defe8c93a0c4e"}
Oct 29 22:57:53.202189 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:53.202097    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:57:53.245743 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:53.245692    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g" podStartSLOduration=2.245678124 podStartE2EDuration="2.245678124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:53.241229936 +0000 UTC m=+908.864140958" watchObservedRunningTime="2026-10-29 22:57:53.245678124 +0000 UTC m=+908.868589145"
Oct 29 22:57:54.716074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:54.716045    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"]
Oct 29 22:57:54.801799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:54.795064    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:54.948854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:54.909228    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"]
Oct 29 22:57:54.948854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:54.913081    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a1d1da\""
Oct 29 22:57:54.948854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:54.913207    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a1d1da-dockercfg-4nt6d\""
Oct 29 22:57:54.975996 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:54.975659    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:54.975996 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:54.975756    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmlzr\" (UniqueName: \"kubernetes.io/projected/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-kube-api-access-vmlzr\") pod \"payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:55.077842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:55.077809    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:55.078027 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:55.077892    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmlzr\" (UniqueName: \"kubernetes.io/projected/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-kube-api-access-vmlzr\") pod \"payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:55.078606 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:55.078585    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:55.101356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:55.101332    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmlzr\" (UniqueName: \"kubernetes.io/projected/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-kube-api-access-vmlzr\") pod \"payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:55.352709 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:55.352625    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:56.410245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:56.410216    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"]
Oct 29 22:57:57.547396 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:57.547348    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5" event={"ID":"361dc423-b463-4373-ae2e-d4f6cf5e7a4c","Type":"ContainerStarted","Data":"1e0b7f7abeae5331009ab6a412ff4073ddf6a6b6e7ecc5a6ef2337f8d4511fe6"}
Oct 29 22:57:57.547989 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:57.547953    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5" event={"ID":"361dc423-b463-4373-ae2e-d4f6cf5e7a4c","Type":"ContainerStarted","Data":"cd203051ca75ba11867b4c5d73ca48c679a6edb8bcfe1fdea0293c71f5445027"}
Oct 29 22:57:57.570805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:57.548350    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:57:57.635196 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:57.635112    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5" podStartSLOduration=3.635084499 podStartE2EDuration="3.635084499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:57.595025945 +0000 UTC m=+913.217936978" watchObservedRunningTime="2026-10-29 22:57:57.635084499 +0000 UTC m=+913.257995595"
Oct 29 22:57:57.817843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:57.807738    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:57.840805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:57.834199    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.091878 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.090608    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.091878 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.091569    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.120228 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.120186    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.161886 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.161840    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.303792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.302930    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.313791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.307840    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.336979 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.336922    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.514797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.509864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.514797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.510267    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.598888 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.598845    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 29 22:57:58.673295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.672242    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.810957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.810849    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.811624 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.811599    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:57:58.843887 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.843848    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.844524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.844501    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:57:58.885422 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:58.885378    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 29 22:57:59.097849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.048408    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 29 22:57:59.210851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.119248    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.210851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.167503    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:57:59.210851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.169730    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.210851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.170142    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:57:59.210851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.192979    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.210851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.193504    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:57:59.314905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.303973    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.314905 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.304527    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:57:59.341538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.335909    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.341538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.336470    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:57:59.341538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.338009    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.341538 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.341000    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:57:59.783815 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.783758    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.784824 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.784793    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.797788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.785853    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.805788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.798578    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:57:59.818788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.806189    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:57:59.823918 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.819077    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:57:59.848791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.816889    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.874798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.849307    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.015985 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.958272    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded"
Oct 29 22:58:00.015985 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.972396    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:00.015985 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:57:59.972467    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.137852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.122552    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 29 22:58:00.137852 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.122496    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.184223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.175881    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.208790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.196040    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.347791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.296179    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:00.347791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.296237    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.347791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.319722    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.348892 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.348846    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.363787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.363732    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.864708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.805613    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.067869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.861422    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.067869 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:01.064097    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.073107 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.862156    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.126849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:00.861664    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.266145 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:01.266032    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.431993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:01.431891    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.520987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:01.458828    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.647791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:01.500524    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.790849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:01.529416    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.790849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:01.566826    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.978917 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:58:01.955268    4827 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.475s"
Oct 29 22:58:02.025937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.023058    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.051858 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.051200    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.051858 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.051652    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:02.051858 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.051698    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.092406 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.066273    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.122824 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.093250    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.125250 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.124537    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.162072 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.162037    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:02.162320 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.162293    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.280272 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.280232    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.319321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.319228    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x" podUID="f90fd47c-4d11-4faa-8eb9-d394960cc2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.435917 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.435871    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.560785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.560727    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.679145 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.667395    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.680508 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.680470    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.805137 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.805100    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.866951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:02.866907    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.056932 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.056829    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.058146 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.058114    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.098760 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.098722    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.139307 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.139257    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 29 22:58:03.139658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.139632    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.142089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.142036    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.301893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.301306    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.479907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.479859    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.669031 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.668987    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.685156 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:58:03.680917    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:58:03.685466 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.685428    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.858009 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.857912    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.875794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.865316    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.875794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.874912    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:03.875794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.874969    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.907787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.895970    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.926788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.919986    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:03.926788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:03.920066    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.067791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.062522    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.067791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.062915    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.103956 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.103869    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.150096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.148868    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.150096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.149555    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.525859 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.517427    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.566793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.530981    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.687790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.669319    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.689782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.688384    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.901090 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.901054    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.915270 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.903424    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.920787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.905597    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:58:04.936931 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:04.936902    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"]
Oct 29 22:58:05.063132 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.063087    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.085855 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.085803    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.097041 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.097009    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"]
Oct 29 22:58:05.097418 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.097398    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.146793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.119594    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:58:05.201842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.176675    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.286187 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.175232    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.383960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.285350    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsz4\" (UniqueName: \"kubernetes.io/projected/cc58ac49-6ca7-4543-a753-dffaab298fdd-kube-api-access-zmsz4\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.384391 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.384366    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.441684 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.441627    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.442642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.403407    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:58:05.442890 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.389651    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:05.574840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.522793    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.599789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.547327    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:58:05.599789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.548810    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.599789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.549922    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.599789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.578866    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsz4\" (UniqueName: \"kubernetes.io/projected/cc58ac49-6ca7-4543-a753-dffaab298fdd-kube-api-access-zmsz4\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.599789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.579046    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.636806 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.631243    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.740546 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.740513    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsz4\" (UniqueName: \"kubernetes.io/projected/cc58ac49-6ca7-4543-a753-dffaab298fdd-kube-api-access-zmsz4\") pod \"maas-api-revoke-keys-e2e-api-a-e4a049-hh565\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.763143 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.763111    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:05.916360 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.916320    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:05.917906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.917136    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.945786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.917169    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.977338 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:05.917604    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:06.069872 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.069828    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:58:06.087053 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.087012    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:58:06.121985 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.121944    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:58:06.187849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.187041    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:58:06.309484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.296870    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:06.309484 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.296940    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:06.317861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.317827    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:06.318364 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.318333    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:06.668914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.668869    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:06.669154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.669130    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:07.031401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.729433    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:58:07.038212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:06.750020    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:58:07.038671 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.013330    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:07.039130 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.013369    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 22:58:07.039432 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.030393    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:07.039708 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.030430    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 22:58:07.098789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.090515    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:58:07.262797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.097182    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:07.344848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.205556    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:07.483790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.482707    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:07.508075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.508032    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:58:07.766792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.758853    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Oct 29 22:58:07.806430 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:07.804176    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.052623 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.052533    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.053532 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.053470    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:08.053729 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.053705    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:58:08.054832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.054709    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.059878 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.055195    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.063509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.063482    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.087800 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:58:08.084971    4827 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.333s"
Oct 29 22:58:08.131001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.130860    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 22:58:08.131001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.130901    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"]
Oct 29 22:58:08.131001 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.130993    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23da404e-fswcw"]
Oct 29 22:58:08.131533 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.131019    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"]
Oct 29 22:58:08.134019 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.133694    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 22:58:08.211489 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.211155    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.297110 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.296262    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.461045 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.461015    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 22:58:08.482358 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.482330    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 22:58:08.506602 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.498974    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 22:58:08.556119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.547864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:08.556119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.551515    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 22:58:08.629450 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.629386    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19792e88-e399-4e25-aa41-79f802825e55" path="/var/lib/kubelet/pods/19792e88-e399-4e25-aa41-79f802825e55/volumes"
Oct 29 22:58:08.645302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.645097    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 22:58:08.645302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.645190    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 22:58:08.645302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.645247    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 22:58:08.645302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.645303    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 22:58:08.645666 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.645359    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 22:58:08.649194 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.649168    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 22:58:08.830744 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:08.830641    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565" event={"ID":"cc58ac49-6ca7-4543-a753-dffaab298fdd","Type":"ContainerStarted","Data":"ebea423650170e1ab33a96420278de13d73cc3cbae5aca9feea5799d28f5d44d"}
Oct 29 22:58:09.928061 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:09.928014    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565" event={"ID":"cc58ac49-6ca7-4543-a753-dffaab298fdd","Type":"ContainerStarted","Data":"2579d31cf059e53b2eac198e63e49f85d64ec21583bf9882e8eb6617849d0659"}
Oct 29 22:58:09.997894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:09.997818    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565" podStartSLOduration=6.997798709 podStartE2EDuration="6.997798709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:09.975704855 +0000 UTC m=+925.598615876" watchObservedRunningTime="2026-10-29 22:58:09.997798709 +0000 UTC m=+925.620709755"
Oct 29 22:58:10.108016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:10.107929    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:58:10.938760 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:10.938727    4827 generic.go:356] "Generic (PLEG): container finished" podID="cc58ac49-6ca7-4543-a753-dffaab298fdd" containerID="2579d31cf059e53b2eac198e63e49f85d64ec21583bf9882e8eb6617849d0659" exitCode=0
Oct 29 22:58:10.939183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:10.938908    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565" event={"ID":"cc58ac49-6ca7-4543-a753-dffaab298fdd","Type":"ContainerDied","Data":"2579d31cf059e53b2eac198e63e49f85d64ec21583bf9882e8eb6617849d0659"}
Oct 29 22:58:11.868843 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.868812    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"]
Oct 29 22:58:11.874459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.874439    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.896040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.891186    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.896040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.891374    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.896040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.891593    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2vg\" (UniqueName: \"kubernetes.io/projected/6a773e44-59ff-4b1a-9729-0ce9420060a4-kube-api-access-2f2vg\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.896040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.894501    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"]
Oct 29 22:58:11.993638 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.993600    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2vg\" (UniqueName: \"kubernetes.io/projected/6a773e44-59ff-4b1a-9729-0ce9420060a4-kube-api-access-2f2vg\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.994124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.993812    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.994124 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.993942    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.994870 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.994846    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:11.996641 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:11.996571    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:12.017514 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.017485    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2vg\" (UniqueName: \"kubernetes.io/projected/6a773e44-59ff-4b1a-9729-0ce9420060a4-kube-api-access-2f2vg\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-g475z\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:12.171428 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.171395    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:12.196390 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.196355    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-xks-ca\") pod \"cc58ac49-6ca7-4543-a753-dffaab298fdd\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") "
Oct 29 22:58:12.196581 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.196420    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc58ac49-6ca7-4543-a753-dffaab298fdd-kube-api-access-zmsz4\" (UniqueName: \"kubernetes.io/projected/cc58ac49-6ca7-4543-a753-dffaab298fdd-kube-api-access-zmsz4\") pod \"cc58ac49-6ca7-4543-a753-dffaab298fdd\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") "
Oct 29 22:58:12.196581 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.196569    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-openshift-ca\") pod \"cc58ac49-6ca7-4543-a753-dffaab298fdd\" (UID: \"cc58ac49-6ca7-4543-a753-dffaab298fdd\") "
Oct 29 22:58:12.198107 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.198072    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-openshift-ca" pod "cc58ac49-6ca7-4543-a753-dffaab298fdd" (UID: "cc58ac49-6ca7-4543-a753-dffaab298fdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:12.199137 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.198913    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-xks-ca" pod "cc58ac49-6ca7-4543-a753-dffaab298fdd" (UID: "cc58ac49-6ca7-4543-a753-dffaab298fdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:12.200951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.200077    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:12.201183 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.201135    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc58ac49-6ca7-4543-a753-dffaab298fdd-kube-api-access-zmsz4" pod "cc58ac49-6ca7-4543-a753-dffaab298fdd" (UID: "cc58ac49-6ca7-4543-a753-dffaab298fdd"). InnerVolumeSpecName "kube-api-access-zmsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:12.346830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.346725    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.346830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.346781    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsz4\" (UniqueName: \"kubernetes.io/projected/cc58ac49-6ca7-4543-a753-dffaab298fdd-kube-api-access-zmsz4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.346830 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.346797    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc58ac49-6ca7-4543-a753-dffaab298fdd-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.691609 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:58:12.691548    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a773e44_59ff_4b1a_9729_0ce9420060a4.slice/crio-42c81ad68d6898d2faeef1d6091cfefe55afb62c14a541e65e534ce090802197 WatchSource:0}: Error finding container 42c81ad68d6898d2faeef1d6091cfefe55afb62c14a541e65e534ce090802197: Status 404 returned error can't find the container with id 42c81ad68d6898d2faeef1d6091cfefe55afb62c14a541e65e534ce090802197
Oct 29 22:58:12.693009 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.692980    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"]
Oct 29 22:58:12.990410 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.990354    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565" event={"ID":"cc58ac49-6ca7-4543-a753-dffaab298fdd","Type":"ContainerDied","Data":"ebea423650170e1ab33a96420278de13d73cc3cbae5aca9feea5799d28f5d44d"}
Oct 29 22:58:12.990410 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.990388    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"
Oct 29 22:58:12.990631 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.990476    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebea423650170e1ab33a96420278de13d73cc3cbae5aca9feea5799d28f5d44d"
Oct 29 22:58:12.993121 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:12.993091    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z" event={"ID":"6a773e44-59ff-4b1a-9729-0ce9420060a4","Type":"ContainerStarted","Data":"42c81ad68d6898d2faeef1d6091cfefe55afb62c14a541e65e534ce090802197"}
Oct 29 22:58:14.116356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:14.116312    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z" event={"ID":"6a773e44-59ff-4b1a-9729-0ce9420060a4","Type":"ContainerStarted","Data":"a517c5048c96d72bd2d566f6c04dedeec2a792bc97aa40596a27161e03b95b50"}
Oct 29 22:58:14.624029 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:14.623994    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"]
Oct 29 22:58:15.131877 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.131839    4827 generic.go:356] "Generic (PLEG): container finished" podID="6a773e44-59ff-4b1a-9729-0ce9420060a4" containerID="a517c5048c96d72bd2d566f6c04dedeec2a792bc97aa40596a27161e03b95b50" exitCode=0
Oct 29 22:58:15.132348 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.131911    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z" event={"ID":"6a773e44-59ff-4b1a-9729-0ce9420060a4","Type":"ContainerDied","Data":"a517c5048c96d72bd2d566f6c04dedeec2a792bc97aa40596a27161e03b95b50"}
Oct 29 22:58:15.363827 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.358746    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:15.445879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.444261    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-xks-ca\") pod \"6a773e44-59ff-4b1a-9729-0ce9420060a4\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") "
Oct 29 22:58:15.445879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.444404    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-openshift-ca\") pod \"6a773e44-59ff-4b1a-9729-0ce9420060a4\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") "
Oct 29 22:58:15.445879 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.444452    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a773e44-59ff-4b1a-9729-0ce9420060a4-kube-api-access-2f2vg\" (UniqueName: \"kubernetes.io/projected/6a773e44-59ff-4b1a-9729-0ce9420060a4-kube-api-access-2f2vg\") pod \"6a773e44-59ff-4b1a-9729-0ce9420060a4\" (UID: \"6a773e44-59ff-4b1a-9729-0ce9420060a4\") "
Oct 29 22:58:15.457529 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.457485    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-openshift-ca" pod "6a773e44-59ff-4b1a-9729-0ce9420060a4" (UID: "6a773e44-59ff-4b1a-9729-0ce9420060a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:15.496486 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.493440    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a773e44-59ff-4b1a-9729-0ce9420060a4-kube-api-access-2f2vg" pod "6a773e44-59ff-4b1a-9729-0ce9420060a4" (UID: "6a773e44-59ff-4b1a-9729-0ce9420060a4"). InnerVolumeSpecName "kube-api-access-2f2vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:15.522752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.522400    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-xks-ca" pod "6a773e44-59ff-4b1a-9729-0ce9420060a4" (UID: "6a773e44-59ff-4b1a-9729-0ce9420060a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:15.555794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.553285    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:15.555794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.553335    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a773e44-59ff-4b1a-9729-0ce9420060a4-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:15.555794 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:15.553352    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2vg\" (UniqueName: \"kubernetes.io/projected/6a773e44-59ff-4b1a-9729-0ce9420060a4-kube-api-access-2f2vg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:16.268462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:16.268431    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"
Oct 29 22:58:16.268983 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:16.268463    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z" event={"ID":"6a773e44-59ff-4b1a-9729-0ce9420060a4","Type":"ContainerDied","Data":"42c81ad68d6898d2faeef1d6091cfefe55afb62c14a541e65e534ce090802197"}
Oct 29 22:58:16.268983 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:16.268540    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42c81ad68d6898d2faeef1d6091cfefe55afb62c14a541e65e534ce090802197"
Oct 29 22:58:17.278527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.278504    4827 generic.go:356] "Generic (PLEG): container finished" podID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerID="4f62da5ddd3f9310faa1c140ef75a3f5e4e43704bfb3ddbcf922bef4882e1c5d" exitCode=0
Oct 29 22:58:17.278918 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.278576    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" event={"ID":"c197150f-64b3-41b9-a0ed-951e3cb3212d","Type":"ContainerDied","Data":"4f62da5ddd3f9310faa1c140ef75a3f5e4e43704bfb3ddbcf922bef4882e1c5d"}
Oct 29 22:58:17.279159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.279141    4827 scope.go:122] "RemoveContainer" containerID="4f62da5ddd3f9310faa1c140ef75a3f5e4e43704bfb3ddbcf922bef4882e1c5d"
Oct 29 22:58:17.280575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.280556    4827 generic.go:356] "Generic (PLEG): container finished" podID="8923c498-8a01-4288-85ae-8f71b92545bf" containerID="6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a" exitCode=0
Oct 29 22:58:17.280658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.280600    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" event={"ID":"8923c498-8a01-4288-85ae-8f71b92545bf","Type":"ContainerDied","Data":"6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a"}
Oct 29 22:58:17.280993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.280974    4827 scope.go:122] "RemoveContainer" containerID="6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a"
Oct 29 22:58:17.281127 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.281101    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:58:17.281185 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.281176    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:58:17.281233 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.281212    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:58:17.282898 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.282867    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:58:17.283002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.282947    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:58:17.283002 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:17.282992    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:58:18.232486 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.232453    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:58:18.320160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.320125    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" event={"ID":"c197150f-64b3-41b9-a0ed-951e3cb3212d","Type":"ContainerStarted","Data":"a0420fef29f16ea9a567ed0e191edd775d2741f83f8b94291269cd462d7515a5"}
Oct 29 22:58:18.322537 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.322510    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" event={"ID":"8923c498-8a01-4288-85ae-8f71b92545bf","Type":"ContainerStarted","Data":"29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900"}
Oct 29 22:58:18.514598 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.514046    4827 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:58:18.514598 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.514147    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 22:58:18.519496 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.519469    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:58:18.904791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.903884    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:58:18.904791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.904056    4827 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:58:18.904791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:18.904112    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:58:19.258837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.248253    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:58:19.513740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.513650    4827 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:58:19.514270 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.513736    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 22:58:19.624808 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.624755    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj" podUID="62509af9-d4c8-4148-aba9-f996a9dff494" containerName="payload-pre-processing" containerID="cri-o://17d92f24c993c140e352f9a6052230c5851bb5f529a3e915ed8da73348726008" gracePeriod=25
Oct 29 22:58:19.647844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.647818    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"]
Oct 29 22:58:19.648571 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.648557    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cc58ac49-6ca7-4543-a753-dffaab298fdd" containerName="revoke-keys"
Oct 29 22:58:19.648618 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.648572    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc58ac49-6ca7-4543-a753-dffaab298fdd" containerName="revoke-keys"
Oct 29 22:58:19.648618 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.648583    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6a773e44-59ff-4b1a-9729-0ce9420060a4" containerName="revoke-keys"
Oct 29 22:58:19.648618 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.648590    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a773e44-59ff-4b1a-9729-0ce9420060a4" containerName="revoke-keys"
Oct 29 22:58:19.648720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.648709    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc58ac49-6ca7-4543-a753-dffaab298fdd" containerName="revoke-keys"
Oct 29 22:58:19.648720 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.648716    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a773e44-59ff-4b1a-9729-0ce9420060a4" containerName="revoke-keys"
Oct 29 22:58:19.652359 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.652340    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:19.710819 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.695705    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0df9b0-dockercfg-4l7jv\""
Oct 29 22:58:19.710819 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.695787    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0df9b0\""
Oct 29 22:58:19.729901 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.729464    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"]
Oct 29 22:58:19.751954 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.745438    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"]
Oct 29 22:58:19.919851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.893146    4827 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:58:19.919851 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.893192    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:58:19.965824 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.950039    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:19.985796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.968249    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"]
Oct 29 22:58:20.121224 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:19.991636    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxsv\" (UniqueName: \"kubernetes.io/projected/103c5747-89fb-46ce-9dde-c8fe9911e997-kube-api-access-hnxsv\") pod \"payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:20.121224 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.108219    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c5747-89fb-46ce-9dde-c8fe9911e997-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:20.299837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.298974    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c5747-89fb-46ce-9dde-c8fe9911e997-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:20.299837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.299102    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/e9fb824d-0d5d-4417-97ef-8aced007417a-kube-api-access-zgqgf\") pod \"payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:20.310799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.306820    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxsv\" (UniqueName: \"kubernetes.io/projected/103c5747-89fb-46ce-9dde-c8fe9911e997-kube-api-access-hnxsv\") pod \"payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:20.379052 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.312389    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c5747-89fb-46ce-9dde-c8fe9911e997-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:20.379364 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.320038    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fb824d-0d5d-4417-97ef-8aced007417a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:20.409790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.388033    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxsv\" (UniqueName: \"kubernetes.io/projected/103c5747-89fb-46ce-9dde-c8fe9911e997-kube-api-access-hnxsv\") pod \"payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:20.533383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.524147    4827 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:58:20.533383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.524206    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 22:58:20.547620 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.547595    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:20.557142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.553541    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fb824d-0d5d-4417-97ef-8aced007417a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:20.557142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.553793    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/e9fb824d-0d5d-4417-97ef-8aced007417a-kube-api-access-zgqgf\") pod \"payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:20.557142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.555137    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fb824d-0d5d-4417-97ef-8aced007417a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:20.623098 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.597394    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/e9fb824d-0d5d-4417-97ef-8aced007417a-kube-api-access-zgqgf\") pod \"payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:20.636800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.636719    4827 generic.go:356] "Generic (PLEG): container finished" podID="62509af9-d4c8-4148-aba9-f996a9dff494" containerID="17d92f24c993c140e352f9a6052230c5851bb5f529a3e915ed8da73348726008" exitCode=0
Oct 29 22:58:20.636992 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.636808    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj" event={"ID":"62509af9-d4c8-4148-aba9-f996a9dff494","Type":"ContainerDied","Data":"17d92f24c993c140e352f9a6052230c5851bb5f529a3e915ed8da73348726008"}
Oct 29 22:58:20.798790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.798078    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:20.929796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.928168    4827 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:58:20.929796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.928227    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:58:20.956630 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:20.956601    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:58:21.061997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.061946    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"]
Oct 29 22:58:21.123583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.093562    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62509af9-d4c8-4148-aba9-f996a9dff494-kube-api-access-vtclt\" (UniqueName: \"kubernetes.io/projected/62509af9-d4c8-4148-aba9-f996a9dff494-kube-api-access-vtclt\") pod \"62509af9-d4c8-4148-aba9-f996a9dff494\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") "
Oct 29 22:58:21.123583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.117844    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62509af9-d4c8-4148-aba9-f996a9dff494-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62509af9-d4c8-4148-aba9-f996a9dff494-plugins-config-volume\") pod \"62509af9-d4c8-4148-aba9-f996a9dff494\" (UID: \"62509af9-d4c8-4148-aba9-f996a9dff494\") "
Oct 29 22:58:21.123583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.120209    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62509af9-d4c8-4148-aba9-f996a9dff494-plugins-config-volume" pod "62509af9-d4c8-4148-aba9-f996a9dff494" (UID: "62509af9-d4c8-4148-aba9-f996a9dff494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:21.123583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.120457    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62509af9-d4c8-4148-aba9-f996a9dff494-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.123583 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.121073    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62509af9-d4c8-4148-aba9-f996a9dff494-kube-api-access-vtclt" pod "62509af9-d4c8-4148-aba9-f996a9dff494" (UID: "62509af9-d4c8-4148-aba9-f996a9dff494"). InnerVolumeSpecName "kube-api-access-vtclt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:21.222795 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.222683    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtclt\" (UniqueName: \"kubernetes.io/projected/62509af9-d4c8-4148-aba9-f996a9dff494-kube-api-access-vtclt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.488439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.488347    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"]
Oct 29 22:58:21.496980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.496891    4827 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:58:21.500932 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.500890    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 22:58:21.532796 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:58:21.525050    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fb824d_0d5d_4417_97ef_8aced007417a.slice/crio-327d0a57bef21dc7b252d9d3390d60ef9494be292a0383261e3ec76d93ad04d0 WatchSource:0}: Error finding container 327d0a57bef21dc7b252d9d3390d60ef9494be292a0383261e3ec76d93ad04d0: Status 404 returned error can't find the container with id 327d0a57bef21dc7b252d9d3390d60ef9494be292a0383261e3ec76d93ad04d0
Oct 29 22:58:21.759665 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.759609    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg" event={"ID":"e9fb824d-0d5d-4417-97ef-8aced007417a","Type":"ContainerStarted","Data":"327d0a57bef21dc7b252d9d3390d60ef9494be292a0383261e3ec76d93ad04d0"}
Oct 29 22:58:21.761336 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.761308    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"
Oct 29 22:58:21.761450 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.761339    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj" event={"ID":"62509af9-d4c8-4148-aba9-f996a9dff494","Type":"ContainerDied","Data":"a45c6b6bea1082b8645c2e1044bfa99e6ea047d91efc5414ed9e74aad4bd6d59"}
Oct 29 22:58:21.761450 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.761398    4827 scope.go:122] "RemoveContainer" containerID="17d92f24c993c140e352f9a6052230c5851bb5f529a3e915ed8da73348726008"
Oct 29 22:58:21.818086 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.818053    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n" event={"ID":"103c5747-89fb-46ce-9dde-c8fe9911e997","Type":"ContainerStarted","Data":"87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d"}
Oct 29 22:58:21.818246 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.818118    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n" event={"ID":"103c5747-89fb-46ce-9dde-c8fe9911e997","Type":"ContainerStarted","Data":"8eec9a61ce92062e3d63f70832656668b0e266cb5ee17de1f989b1d84f4584ae"}
Oct 29 22:58:21.818483 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.818469    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:21.859276 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.859235    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n" podStartSLOduration=2.859222494 podStartE2EDuration="2.859222494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:21.85837444 +0000 UTC m=+937.481285458" watchObservedRunningTime="2026-10-29 22:58:21.859222494 +0000 UTC m=+937.482133513"
Oct 29 22:58:21.897804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.892323    4827 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:58:21.897804 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.892378    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:58:21.927314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.927286    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"]
Oct 29 22:58:21.936883 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:21.936856    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e4a049-798ff57fbc-ktvsj"]
Oct 29 22:58:22.521062 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:22.521022    4827 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:58:22.521250 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:22.521096    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 22:58:22.538049 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:22.538005    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62509af9-d4c8-4148-aba9-f996a9dff494" path="/var/lib/kubelet/pods/62509af9-d4c8-4148-aba9-f996a9dff494/volumes"
Oct 29 22:58:22.896981 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:22.894007    4827 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:58:22.896981 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:22.894067    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:58:22.958856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:22.955145    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg" event={"ID":"e9fb824d-0d5d-4417-97ef-8aced007417a","Type":"ContainerStarted","Data":"5ac499dd5f5eff583184484fc19fd27ee27af4a4a276f9af30d7c2ba6a6b9ed8"}
Oct 29 22:58:22.958856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:22.955241    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:23.542818 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:23.537184    4827 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:58:23.542818 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:23.537241    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 22:58:23.959846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:23.959420    4827 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:58:23.959846 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:23.959476    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:58:24.205914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:24.205867    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.513198 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:24.513151    4827 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:58:24.525798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:24.513445    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 22:58:24.896294 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:24.896257    4827 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:58:24.896581 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:24.896553    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:58:25.504155 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:25.504118    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:58:25.510145 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:25.510115    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 22:58:25.566136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:25.566078    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg" podStartSLOduration=6.566060141 podStartE2EDuration="6.566060141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:23.046198482 +0000 UTC m=+938.669109503" watchObservedRunningTime="2026-10-29 22:58:25.566060141 +0000 UTC m=+941.188971163"
Oct 29 22:58:25.972161 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:25.972129    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:58:25.978232 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:25.978201    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:58:25.983480 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:25.983451    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 22:58:27.185463 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.185396    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"]
Oct 29 22:58:27.186182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.186163    4827 cpu_manager.go:414] "containerMap: removing container" podUID="62509af9-d4c8-4148-aba9-f996a9dff494" containerName="payload-pre-processing"
Oct 29 22:58:27.186182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.186180    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62509af9-d4c8-4148-aba9-f996a9dff494" containerName="payload-pre-processing"
Oct 29 22:58:27.186355 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.186304    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="62509af9-d4c8-4148-aba9-f996a9dff494" containerName="payload-pre-processing"
Oct 29 22:58:27.193012 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.192983    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.211709 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.211679    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"]
Oct 29 22:58:27.216104 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.216080    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a1d1da-serving-cert\""
Oct 29 22:58:27.216914 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.216893    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xrwtg\""
Oct 29 22:58:27.216996 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.216977    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:58:27.326142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.326112    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhqs6\" (UniqueName: \"kubernetes.io/projected/e8bf8095-b37f-4d76-9572-96e1cb459a80-kube-api-access-vhqs6\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.326327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.326165    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.326327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.326235    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf8095-b37f-4d76-9572-96e1cb459a80-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.326501 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.326481    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-metrics-certs\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.449841 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.429968    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqs6\" (UniqueName: \"kubernetes.io/projected/e8bf8095-b37f-4d76-9572-96e1cb459a80-kube-api-access-vhqs6\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.450171 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.450130    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.450323 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.450220    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf8095-b37f-4d76-9572-96e1cb459a80-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.450503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.450485    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-metrics-certs\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.498047 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.496323    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqs6\" (UniqueName: \"kubernetes.io/projected/e8bf8095-b37f-4d76-9572-96e1cb459a80-kube-api-access-vhqs6\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.502114 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.502080    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-metrics-certs\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.512371 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.512337    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.565787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.548328    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf8095-b37f-4d76-9572-96e1cb459a80-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:27.826440 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:27.826358    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:28.146597 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:58:28.146566    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bf8095_b37f_4d76_9572_96e1cb459a80.slice/crio-c1289fa1f24d3c621b4c692627194ea627650ab6b13c0bb796343a106bf3d7dd WatchSource:0}: Error finding container c1289fa1f24d3c621b4c692627194ea627650ab6b13c0bb796343a106bf3d7dd: Status 404 returned error can't find the container with id c1289fa1f24d3c621b4c692627194ea627650ab6b13c0bb796343a106bf3d7dd
Oct 29 22:58:28.147283 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.147258    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"]
Oct 29 22:58:28.148274 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.148257    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:58:28.187854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.187810    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9" event={"ID":"e8bf8095-b37f-4d76-9572-96e1cb459a80","Type":"ContainerStarted","Data":"c1289fa1f24d3c621b4c692627194ea627650ab6b13c0bb796343a106bf3d7dd"}
Oct 29 22:58:28.714082 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.714050    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"]
Oct 29 22:58:28.720409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.720386    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.739209 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.739184    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"]
Oct 29 22:58:28.813010 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.812979    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:28.813167 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.813150    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:58:28.814040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.814015    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.814212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.814196    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dws8x\" (UniqueName: \"kubernetes.io/projected/b735b8dd-105e-40c8-96eb-b5af28d4d387-kube-api-access-dws8x\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.814417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.814397    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.918040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.916062    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.918040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.916204    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.918040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.916333    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dws8x\" (UniqueName: \"kubernetes.io/projected/b735b8dd-105e-40c8-96eb-b5af28d4d387-kube-api-access-dws8x\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.918040 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.917352    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.927153 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.927126    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:28.995585 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:28.995508    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dws8x\" (UniqueName: \"kubernetes.io/projected/b735b8dd-105e-40c8-96eb-b5af28d4d387-kube-api-access-dws8x\") pod \"maas-api-revoke-sub-1c9f72b8-fh5fw\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:29.041966 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:29.041938    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:29.424811 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:29.424040    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"]
Oct 29 22:58:30.373448 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:30.373412    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9" event={"ID":"e8bf8095-b37f-4d76-9572-96e1cb459a80","Type":"ContainerStarted","Data":"22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa"}
Oct 29 22:58:30.373642 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:30.373514    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:30.375519 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:30.375495    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw" event={"ID":"b735b8dd-105e-40c8-96eb-b5af28d4d387","Type":"ContainerStarted","Data":"15ed3cff21850bffc7e9940b440df72cc9e16ba3108ece7ac769e4c6ddce01c7"}
Oct 29 22:58:30.375651 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:30.375544    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw" event={"ID":"b735b8dd-105e-40c8-96eb-b5af28d4d387","Type":"ContainerStarted","Data":"94c42d746ca836a2f65a997ff8ee49d71832f9492ef979f35ecf216eb069b86f"}
Oct 29 22:58:30.420196 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:30.420142    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9" podStartSLOduration=1.5435800880000001 podStartE2EDuration="3.420128653s" totalImagesPullingTime="1.876548565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:30.414338989 +0000 UTC m=+946.037250009" watchObservedRunningTime="2026-10-29 22:58:30.420128653 +0000 UTC m=+946.043039672"
Oct 29 22:58:30.454438 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:30.454395    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw" podStartSLOduration=2.454381424 podStartE2EDuration="2.454381424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:30.453524417 +0000 UTC m=+946.076435437" watchObservedRunningTime="2026-10-29 22:58:30.454381424 +0000 UTC m=+946.077292500"
Oct 29 22:58:31.386393 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:31.386354    4827 generic.go:356] "Generic (PLEG): container finished" podID="b735b8dd-105e-40c8-96eb-b5af28d4d387" containerID="15ed3cff21850bffc7e9940b440df72cc9e16ba3108ece7ac769e4c6ddce01c7" exitCode=0
Oct 29 22:58:31.386592 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:31.386418    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw" event={"ID":"b735b8dd-105e-40c8-96eb-b5af28d4d387","Type":"ContainerDied","Data":"15ed3cff21850bffc7e9940b440df72cc9e16ba3108ece7ac769e4c6ddce01c7"}
Oct 29 22:58:32.577949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.577922    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:32.653920 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.653838    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b735b8dd-105e-40c8-96eb-b5af28d4d387-kube-api-access-dws8x\" (UniqueName: \"kubernetes.io/projected/b735b8dd-105e-40c8-96eb-b5af28d4d387-kube-api-access-dws8x\") pod \"b735b8dd-105e-40c8-96eb-b5af28d4d387\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") "
Oct 29 22:58:32.654168 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.654141    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-xks-ca\") pod \"b735b8dd-105e-40c8-96eb-b5af28d4d387\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") "
Oct 29 22:58:32.654360 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.654188    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-openshift-ca\") pod \"b735b8dd-105e-40c8-96eb-b5af28d4d387\" (UID: \"b735b8dd-105e-40c8-96eb-b5af28d4d387\") "
Oct 29 22:58:32.654909 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.654884    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-openshift-ca" pod "b735b8dd-105e-40c8-96eb-b5af28d4d387" (UID: "b735b8dd-105e-40c8-96eb-b5af28d4d387"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:32.655159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.655135    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:32.656625 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.656601    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-xks-ca" pod "b735b8dd-105e-40c8-96eb-b5af28d4d387" (UID: "b735b8dd-105e-40c8-96eb-b5af28d4d387"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:32.656732 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.656647    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b735b8dd-105e-40c8-96eb-b5af28d4d387-kube-api-access-dws8x" pod "b735b8dd-105e-40c8-96eb-b5af28d4d387" (UID: "b735b8dd-105e-40c8-96eb-b5af28d4d387"). InnerVolumeSpecName "kube-api-access-dws8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:32.757095 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.757065    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735b8dd-105e-40c8-96eb-b5af28d4d387-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:32.757095 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.757092    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dws8x\" (UniqueName: \"kubernetes.io/projected/b735b8dd-105e-40c8-96eb-b5af28d4d387-kube-api-access-dws8x\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:32.825608 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:58:32.825574    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:58:32.957972 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:32.957890    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 22:58:33.448791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:33.448686    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw" event={"ID":"b735b8dd-105e-40c8-96eb-b5af28d4d387","Type":"ContainerDied","Data":"94c42d746ca836a2f65a997ff8ee49d71832f9492ef979f35ecf216eb069b86f"}
Oct 29 22:58:33.448994 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:33.448815    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94c42d746ca836a2f65a997ff8ee49d71832f9492ef979f35ecf216eb069b86f"
Oct 29 22:58:33.448994 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:33.448906    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"
Oct 29 22:58:35.172717 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:35.172687    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 22:58:36.337548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.337515    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"]
Oct 29 22:58:36.338418 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.338396    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b735b8dd-105e-40c8-96eb-b5af28d4d387" containerName="revoke-keys"
Oct 29 22:58:36.338418 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.338415    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b735b8dd-105e-40c8-96eb-b5af28d4d387" containerName="revoke-keys"
Oct 29 22:58:36.338587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.338535    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b735b8dd-105e-40c8-96eb-b5af28d4d387" containerName="revoke-keys"
Oct 29 22:58:36.343215 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.343197    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.362751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.362726    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"]
Oct 29 22:58:36.367085 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.367066    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:36.367314 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.367293    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:58:36.392673 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.392650    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 22:58:36.703789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.700159    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.703789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.700316    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvg7g\" (UniqueName: \"kubernetes.io/projected/4ab71a29-b7b6-4854-ba77-4c1be33ec750-kube-api-access-qvg7g\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.703789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.700559    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.852786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.848044    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.852786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.848295    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.852786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.848395    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvg7g\" (UniqueName: \"kubernetes.io/projected/4ab71a29-b7b6-4854-ba77-4c1be33ec750-kube-api-access-qvg7g\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.887854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.866019    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.912833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.910824    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvg7g\" (UniqueName: \"kubernetes.io/projected/4ab71a29-b7b6-4854-ba77-4c1be33ec750-kube-api-access-qvg7g\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.932404 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.932368    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a37a29-gntd2\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:36.960813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:36.960718    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:37.273440 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:58:37.273408    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab71a29_b7b6_4854_ba77_4c1be33ec750.slice/crio-bc1cb8810b5f55ac685293d5d0e74ffddbc9aa361c542a46ebd486fdecdce7c7 WatchSource:0}: Error finding container bc1cb8810b5f55ac685293d5d0e74ffddbc9aa361c542a46ebd486fdecdce7c7: Status 404 returned error can't find the container with id bc1cb8810b5f55ac685293d5d0e74ffddbc9aa361c542a46ebd486fdecdce7c7
Oct 29 22:58:37.275059 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:37.275035    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"]
Oct 29 22:58:37.855479 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:37.855447    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2" event={"ID":"4ab71a29-b7b6-4854-ba77-4c1be33ec750","Type":"ContainerStarted","Data":"4c669f16cc029e44fbff08a96ef365b629bce409852973990146b3a0808e0410"}
Oct 29 22:58:37.856075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:37.855499    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2" event={"ID":"4ab71a29-b7b6-4854-ba77-4c1be33ec750","Type":"ContainerStarted","Data":"bc1cb8810b5f55ac685293d5d0e74ffddbc9aa361c542a46ebd486fdecdce7c7"}
Oct 29 22:58:37.901207 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:37.901106    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2" podStartSLOduration=1.901091622 podStartE2EDuration="1.901091622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:37.894779399 +0000 UTC m=+953.517690412" watchObservedRunningTime="2026-10-29 22:58:37.901091622 +0000 UTC m=+953.524002670"
Oct 29 22:58:38.032858 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:38.032825    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"]
Oct 29 22:58:38.038269 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:38.038240    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688e0e11-rx2cw"]
Oct 29 22:58:38.491226 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:38.491149    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080b5369-dcb9-491c-968b-e209b8b6a807" path="/var/lib/kubelet/pods/080b5369-dcb9-491c-968b-e209b8b6a807/volumes"
Oct 29 22:58:38.867576 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:38.867488    4827 generic.go:356] "Generic (PLEG): container finished" podID="4ab71a29-b7b6-4854-ba77-4c1be33ec750" containerID="4c669f16cc029e44fbff08a96ef365b629bce409852973990146b3a0808e0410" exitCode=0
Oct 29 22:58:38.867998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:38.867574    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2" event={"ID":"4ab71a29-b7b6-4854-ba77-4c1be33ec750","Type":"ContainerDied","Data":"4c669f16cc029e44fbff08a96ef365b629bce409852973990146b3a0808e0410"}
Oct 29 22:58:40.055541 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.055509    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:40.161980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.161907    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-openshift-ca\") pod \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") "
Oct 29 22:58:40.161980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.161953    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-xks-ca\") pod \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") "
Oct 29 22:58:40.162212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.162044    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab71a29-b7b6-4854-ba77-4c1be33ec750-kube-api-access-qvg7g\" (UniqueName: \"kubernetes.io/projected/4ab71a29-b7b6-4854-ba77-4c1be33ec750-kube-api-access-qvg7g\") pod \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\" (UID: \"4ab71a29-b7b6-4854-ba77-4c1be33ec750\") "
Oct 29 22:58:40.162335 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.162308    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-openshift-ca" pod "4ab71a29-b7b6-4854-ba77-4c1be33ec750" (UID: "4ab71a29-b7b6-4854-ba77-4c1be33ec750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:40.162664 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.162650    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:40.164259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.164234    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-xks-ca" pod "4ab71a29-b7b6-4854-ba77-4c1be33ec750" (UID: "4ab71a29-b7b6-4854-ba77-4c1be33ec750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:40.164259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.164239    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab71a29-b7b6-4854-ba77-4c1be33ec750-kube-api-access-qvg7g" pod "4ab71a29-b7b6-4854-ba77-4c1be33ec750" (UID: "4ab71a29-b7b6-4854-ba77-4c1be33ec750"). InnerVolumeSpecName "kube-api-access-qvg7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:40.264701 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.264672    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab71a29-b7b6-4854-ba77-4c1be33ec750-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:40.264701 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.264700    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvg7g\" (UniqueName: \"kubernetes.io/projected/4ab71a29-b7b6-4854-ba77-4c1be33ec750-kube-api-access-qvg7g\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:40.935995 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.932989    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2" event={"ID":"4ab71a29-b7b6-4854-ba77-4c1be33ec750","Type":"ContainerDied","Data":"bc1cb8810b5f55ac685293d5d0e74ffddbc9aa361c542a46ebd486fdecdce7c7"}
Oct 29 22:58:40.935995 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.933049    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc1cb8810b5f55ac685293d5d0e74ffddbc9aa361c542a46ebd486fdecdce7c7"
Oct 29 22:58:40.935995 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:40.933113    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"
Oct 29 22:58:42.185494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.185455    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"]
Oct 29 22:58:42.186346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.186323    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4ab71a29-b7b6-4854-ba77-4c1be33ec750" containerName="revoke-keys"
Oct 29 22:58:42.186346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.186344    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab71a29-b7b6-4854-ba77-4c1be33ec750" containerName="revoke-keys"
Oct 29 22:58:42.186509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.186496    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab71a29-b7b6-4854-ba77-4c1be33ec750" containerName="revoke-keys"
Oct 29 22:58:42.190997 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.190975    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.210961 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.210935    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0df9b0-serving-cert\""
Oct 29 22:58:42.212422 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.212392    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5g97\" (UniqueName: \"kubernetes.io/projected/02a2c601-a75e-439a-9400-9ee9e86c8c6a-kube-api-access-t5g97\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.212907 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.212882    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-metrics-certs\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.213293 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.213270    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-maas-api-tls\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.213478 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.213451    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a2c601-a75e-439a-9400-9ee9e86c8c6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.221814 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.216061    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"]
Oct 29 22:58:42.315264 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.315228    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5g97\" (UniqueName: \"kubernetes.io/projected/02a2c601-a75e-439a-9400-9ee9e86c8c6a-kube-api-access-t5g97\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.315456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.315272    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-metrics-certs\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.315456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.315411    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-maas-api-tls\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.315456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.315441    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a2c601-a75e-439a-9400-9ee9e86c8c6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.316211 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.316187    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a2c601-a75e-439a-9400-9ee9e86c8c6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.317802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.317759    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-maas-api-tls\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.317890 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.317761    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-metrics-certs\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.339799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.339761    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5g97\" (UniqueName: \"kubernetes.io/projected/02a2c601-a75e-439a-9400-9ee9e86c8c6a-kube-api-access-t5g97\") pod \"maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.565006 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.563392    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:42.928377 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:58:42.928345    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a2c601_a75e_439a_9400_9ee9e86c8c6a.slice/crio-8f9d48e5876648f57c5017d24539d8191038e8ee292ea593507e1912eb6b17a4 WatchSource:0}: Error finding container 8f9d48e5876648f57c5017d24539d8191038e8ee292ea593507e1912eb6b17a4: Status 404 returned error can't find the container with id 8f9d48e5876648f57c5017d24539d8191038e8ee292ea593507e1912eb6b17a4
Oct 29 22:58:42.930089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.930066    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"]
Oct 29 22:58:42.951627 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:42.951598    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm" event={"ID":"02a2c601-a75e-439a-9400-9ee9e86c8c6a","Type":"ContainerStarted","Data":"8f9d48e5876648f57c5017d24539d8191038e8ee292ea593507e1912eb6b17a4"}
Oct 29 22:58:43.961998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:43.961956    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm" event={"ID":"02a2c601-a75e-439a-9400-9ee9e86c8c6a","Type":"ContainerStarted","Data":"a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67"}
Oct 29 22:58:43.962445 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:43.962047    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:44.011876 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:44.011826    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm" podStartSLOduration=1.5373389899999998 podStartE2EDuration="2.011811474s" totalImagesPullingTime="474.472484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:44.003815577 +0000 UTC m=+959.626726617" watchObservedRunningTime="2026-10-29 22:58:44.011811474 +0000 UTC m=+959.634722494"
Oct 29 22:58:45.072190 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:45.072160    4827 scope.go:122] "RemoveContainer" containerID="f942fa2ea6b204d1c41c14a99a3f1d02d1f94ebe2996a20a23517b0c20782eea"
Oct 29 22:58:45.084146 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:45.084124    4827 scope.go:122] "RemoveContainer" containerID="64a31dfe3cd444c74d1eb7b17d38d6911af73beb05972996773de4471834334c"
Oct 29 22:58:45.098297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:45.097467    4827 scope.go:122] "RemoveContainer" containerID="96b864d94ae9e4588b2459a0a67d4cd5799e351b4cbf37dcdc116e5dd583a064"
Oct 29 22:58:45.108154 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:45.108135    4827 scope.go:122] "RemoveContainer" containerID="97651af2869f98458c739f055782249ceed753a7bd3f16d9c943be64797341e8"
Oct 29 22:58:45.117632 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:45.117610    4827 scope.go:122] "RemoveContainer" containerID="1466394bd06ec48b079632ced2a94bbd85a151801aff81e43bbdff7f77dc00c9"
Oct 29 22:58:46.842690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:46.842652    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"]
Oct 29 22:58:49.977386 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:49.977354    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 22:58:50.569788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:50.569163    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"]
Oct 29 22:58:51.536817 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:51.528074    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"]
Oct 29 22:58:51.579799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:51.576985    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e4a049-hh565"]
Oct 29 22:58:51.851442 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:51.847829    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz" podUID="57d893fd-9b31-4e44-9905-48b6fd168512" containerName="payload-pre-processing" containerID="cri-o://c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425" gracePeriod=25
Oct 29 22:58:52.272838 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.272812    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:58:52.285316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.285282    4827 generic.go:356] "Generic (PLEG): container finished" podID="57d893fd-9b31-4e44-9905-48b6fd168512" containerID="c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425" exitCode=0
Oct 29 22:58:52.285469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.285318    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"
Oct 29 22:58:52.285469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.285355    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz" event={"ID":"57d893fd-9b31-4e44-9905-48b6fd168512","Type":"ContainerDied","Data":"c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425"}
Oct 29 22:58:52.285469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.285438    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz" event={"ID":"57d893fd-9b31-4e44-9905-48b6fd168512","Type":"ContainerDied","Data":"8e327662c80c197e6f23061daee69dbbbc7b77f8d7d0c569e384076ea054ef6c"}
Oct 29 22:58:52.285469 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.285461    4827 scope.go:122] "RemoveContainer" containerID="c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425"
Oct 29 22:58:52.299044 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.299018    4827 scope.go:122] "RemoveContainer" containerID="c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425"
Oct 29 22:58:52.299353 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:58:52.299329    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425\": container with ID starting with c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425 not found: ID does not exist" containerID="c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425"
Oct 29 22:58:52.299456 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.299363    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425"} err="failed to get container status \"c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425\": rpc error: code = NotFound desc = could not find container \"c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425\": container with ID starting with c92492dc67124ba95e99f7974ef4a1b870cb552fd82e57d255c5909b181ee425 not found: ID does not exist"
Oct 29 22:58:52.362337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.359907    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d893fd-9b31-4e44-9905-48b6fd168512-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d893fd-9b31-4e44-9905-48b6fd168512-plugins-config-volume\") pod \"57d893fd-9b31-4e44-9905-48b6fd168512\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") "
Oct 29 22:58:52.362337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.360161    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d893fd-9b31-4e44-9905-48b6fd168512-kube-api-access-8vkl9\" (UniqueName: \"kubernetes.io/projected/57d893fd-9b31-4e44-9905-48b6fd168512-kube-api-access-8vkl9\") pod \"57d893fd-9b31-4e44-9905-48b6fd168512\" (UID: \"57d893fd-9b31-4e44-9905-48b6fd168512\") "
Oct 29 22:58:52.362337 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.361721    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d893fd-9b31-4e44-9905-48b6fd168512-plugins-config-volume" pod "57d893fd-9b31-4e44-9905-48b6fd168512" (UID: "57d893fd-9b31-4e44-9905-48b6fd168512"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:52.365500 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.364443    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d893fd-9b31-4e44-9905-48b6fd168512-kube-api-access-8vkl9" pod "57d893fd-9b31-4e44-9905-48b6fd168512" (UID: "57d893fd-9b31-4e44-9905-48b6fd168512"). InnerVolumeSpecName "kube-api-access-8vkl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:52.462327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.462274    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d893fd-9b31-4e44-9905-48b6fd168512-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:52.462327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.462326    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vkl9\" (UniqueName: \"kubernetes.io/projected/57d893fd-9b31-4e44-9905-48b6fd168512-kube-api-access-8vkl9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:52.492575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.492543    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc58ac49-6ca7-4543-a753-dffaab298fdd" path="/var/lib/kubelet/pods/cc58ac49-6ca7-4543-a753-dffaab298fdd/volumes"
Oct 29 22:58:52.628594 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.628557    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"]
Oct 29 22:58:52.641254 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:52.641193    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a37a29-78bffdc4bc-bvnjz"]
Oct 29 22:58:54.492079 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:54.492044    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d893fd-9b31-4e44-9905-48b6fd168512" path="/var/lib/kubelet/pods/57d893fd-9b31-4e44-9905-48b6fd168512/volumes"
Oct 29 22:58:55.570216 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:55.570172    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g" podUID="dbe98308-7569-4c57-80c2-83df2ee955e6" containerName="payload-processing" containerID="cri-o://ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480" gracePeriod=25
Oct 29 22:58:55.874125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:55.874096    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:58:55.944379 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:55.944343    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe98308-7569-4c57-80c2-83df2ee955e6-kube-api-access-fhwrd\" (UniqueName: \"kubernetes.io/projected/dbe98308-7569-4c57-80c2-83df2ee955e6-kube-api-access-fhwrd\") pod \"dbe98308-7569-4c57-80c2-83df2ee955e6\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") "
Oct 29 22:58:55.944813 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:55.944618    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe98308-7569-4c57-80c2-83df2ee955e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe98308-7569-4c57-80c2-83df2ee955e6-plugins-config-volume\") pod \"dbe98308-7569-4c57-80c2-83df2ee955e6\" (UID: \"dbe98308-7569-4c57-80c2-83df2ee955e6\") "
Oct 29 22:58:55.945167 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:55.945136    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe98308-7569-4c57-80c2-83df2ee955e6-plugins-config-volume" pod "dbe98308-7569-4c57-80c2-83df2ee955e6" (UID: "dbe98308-7569-4c57-80c2-83df2ee955e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:55.946058 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:55.946034    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe98308-7569-4c57-80c2-83df2ee955e6-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:55.947213 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:55.947183    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe98308-7569-4c57-80c2-83df2ee955e6-kube-api-access-fhwrd" pod "dbe98308-7569-4c57-80c2-83df2ee955e6" (UID: "dbe98308-7569-4c57-80c2-83df2ee955e6"). InnerVolumeSpecName "kube-api-access-fhwrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:56.048446 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.048416    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhwrd\" (UniqueName: \"kubernetes.io/projected/dbe98308-7569-4c57-80c2-83df2ee955e6-kube-api-access-fhwrd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:58:56.369849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.369819    4827 generic.go:356] "Generic (PLEG): container finished" podID="dbe98308-7569-4c57-80c2-83df2ee955e6" containerID="ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480" exitCode=0
Oct 29 22:58:56.370016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.369863    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"
Oct 29 22:58:56.370016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.369909    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g" event={"ID":"dbe98308-7569-4c57-80c2-83df2ee955e6","Type":"ContainerDied","Data":"ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480"}
Oct 29 22:58:56.370016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.369963    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g" event={"ID":"dbe98308-7569-4c57-80c2-83df2ee955e6","Type":"ContainerDied","Data":"8fd0d84a202e5bd1e99918356257dc6be857bf99c0fb637cd86defe8c93a0c4e"}
Oct 29 22:58:56.370016 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.369987    4827 scope.go:122] "RemoveContainer" containerID="ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480"
Oct 29 22:58:56.382080 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.382063    4827 scope.go:122] "RemoveContainer" containerID="ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480"
Oct 29 22:58:56.382352 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:58:56.382335    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480\": container with ID starting with ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480 not found: ID does not exist" containerID="ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480"
Oct 29 22:58:56.382401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.382360    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480"} err="failed to get container status \"ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480\": rpc error: code = NotFound desc = could not find container \"ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480\": container with ID starting with ff4a92c8f0c2d8b628d321428b27b3a7497aab72b99d9eaf3a26e6e30136c480 not found: ID does not exist"
Oct 29 22:58:56.424709 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.424681    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"]
Oct 29 22:58:56.429752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.429732    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-5fbddfc8fb-dd57g"]
Oct 29 22:58:56.490515 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:56.490484    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe98308-7569-4c57-80c2-83df2ee955e6" path="/var/lib/kubelet/pods/dbe98308-7569-4c57-80c2-83df2ee955e6/volumes"
Oct 29 22:58:58.364266 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:58.364236    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"]
Oct 29 22:58:58.364755 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:58.364454    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy" containerID="cri-o://2f8ca54a5e0481d05629c9b7710ef29ea315868b0c7afa0be1908e1b61a826d1" gracePeriod=30
Oct 29 22:58:58.373212 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:58.373186    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"]
Oct 29 22:58:58.373420 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:58.373399    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy" containerID="cri-o://d814cbe1adbaffa35e08107787be07f3102568d36db71434c1e4af8967507da7" gracePeriod=30
Oct 29 22:58:59.136987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.136956    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"]
Oct 29 22:58:59.194625 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.194589    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"]
Oct 29 22:58:59.200419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.200388    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-g475z"]
Oct 29 22:58:59.584888 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.584798    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"]
Oct 29 22:58:59.586648 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.586618    4827 cpu_manager.go:414] "containerMap: removing container" podUID="dbe98308-7569-4c57-80c2-83df2ee955e6" containerName="payload-processing"
Oct 29 22:58:59.586862 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.586843    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe98308-7569-4c57-80c2-83df2ee955e6" containerName="payload-processing"
Oct 29 22:58:59.586990 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.586976    4827 cpu_manager.go:414] "containerMap: removing container" podUID="57d893fd-9b31-4e44-9905-48b6fd168512" containerName="payload-pre-processing"
Oct 29 22:58:59.587088 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.587074    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d893fd-9b31-4e44-9905-48b6fd168512" containerName="payload-pre-processing"
Oct 29 22:58:59.587390 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.587372    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d893fd-9b31-4e44-9905-48b6fd168512" containerName="payload-pre-processing"
Oct 29 22:58:59.587532 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.587521    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe98308-7569-4c57-80c2-83df2ee955e6" containerName="payload-processing"
Oct 29 22:58:59.626787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.620521    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:58:59.664790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.647207    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:59.664790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.648101    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:58:59.864417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.829021    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmc7b\" (UniqueName: \"kubernetes.io/projected/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-kube-api-access-gmc7b\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:58:59.864417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.829080    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:58:59.864417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.834023    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"]
Oct 29 22:58:59.898849 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:58:59.881307    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.026179 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.026148    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmc7b\" (UniqueName: \"kubernetes.io/projected/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-kube-api-access-gmc7b\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.026381 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.026199    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.026458 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.026407    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.027421 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.027398    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.042937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.042795    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.066585 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.066379    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmc7b\" (UniqueName: \"kubernetes.io/projected/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-kube-api-access-gmc7b\") pod \"maas-api-revoke-keys-e2e-api-b-27d16a-gvm74\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.182650 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.182623    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:00.642622 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:00.641838    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a773e44-59ff-4b1a-9729-0ce9420060a4" path="/var/lib/kubelet/pods/6a773e44-59ff-4b1a-9729-0ce9420060a4/volumes"
Oct 29 22:59:01.092196 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:01.092035    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5757d0ff_f91b_4b0f_bd12_bfca8bd6952c.slice/crio-948b7ae8bf6c692118ec46d42ca7a12934369993a33b0baec8e4c67488d48ae6 WatchSource:0}: Error finding container 948b7ae8bf6c692118ec46d42ca7a12934369993a33b0baec8e4c67488d48ae6: Status 404 returned error can't find the container with id 948b7ae8bf6c692118ec46d42ca7a12934369993a33b0baec8e4c67488d48ae6
Oct 29 22:59:01.095229 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:01.095199    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"]
Oct 29 22:59:01.753800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:01.752388    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74" event={"ID":"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c","Type":"ContainerStarted","Data":"948b7ae8bf6c692118ec46d42ca7a12934369993a33b0baec8e4c67488d48ae6"}
Oct 29 22:59:02.762490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:02.762458    4827 generic.go:356] "Generic (PLEG): container finished" podID="5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" containerID="efc73469de74bef1ea4bbe9bd6b0559b14707f51359e2fe2fda037c9eae8f430" exitCode=0
Oct 29 22:59:02.785005 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:02.762542    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74" event={"ID":"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c","Type":"ContainerDied","Data":"efc73469de74bef1ea4bbe9bd6b0559b14707f51359e2fe2fda037c9eae8f430"}
Oct 29 22:59:02.851093 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:59:02.851008    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:59:03.848504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:03.848167    4827 generic.go:356] "Generic (PLEG): container finished" podID="aaabce08-691b-4828-8810-c426708e784e" containerID="2f8ca54a5e0481d05629c9b7710ef29ea315868b0c7afa0be1908e1b61a826d1" exitCode=0
Oct 29 22:59:03.848504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:03.848373    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" event={"ID":"aaabce08-691b-4828-8810-c426708e784e","Type":"ContainerDied","Data":"2f8ca54a5e0481d05629c9b7710ef29ea315868b0c7afa0be1908e1b61a826d1"}
Oct 29 22:59:03.848504 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:03.848421    4827 scope.go:122] "RemoveContainer" containerID="30e262dfc9ac5a7e450533f7af75562522ff03aa7d7d22892ddcf7705bfa3713"
Oct 29 22:59:03.852102 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:03.851998    4827 generic.go:356] "Generic (PLEG): container finished" podID="61d297af-035a-421e-bf23-38d878371101" containerID="d814cbe1adbaffa35e08107787be07f3102568d36db71434c1e4af8967507da7" exitCode=0
Oct 29 22:59:03.852316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:03.852291    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" event={"ID":"61d297af-035a-421e-bf23-38d878371101","Type":"ContainerDied","Data":"d814cbe1adbaffa35e08107787be07f3102568d36db71434c1e4af8967507da7"}
Oct 29 22:59:04.136109 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.136082    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:59:04.137833 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.137757    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5" podUID="361dc423-b463-4373-ae2e-d4f6cf5e7a4c" containerName="payload-processing" containerID="cri-o://1e0b7f7abeae5331009ab6a412ff4073ddf6a6b6e7ecc5a6ef2337f8d4511fe6" gracePeriod=25
Oct 29 22:59:04.148909 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.148888    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-certs\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.149387 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.149369    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-kube-api-access-vxfnz\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-kube-api-access-vxfnz\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.149854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.149838    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-envoy\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.150204 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.150192    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-data\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.150740 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.150720    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-socket\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.151168 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.151156    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-istio-token\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-istio-token\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.151508 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.151489    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d297af-035a-421e-bf23-38d878371101-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d297af-035a-421e-bf23-38d878371101-istio-podinfo\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.151949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.149258    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-certs" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.151949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.150533    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-data" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.151949 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.151026    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-socket" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.152259 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.151900    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d297af-035a-421e-bf23-38d878371101-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d297af-035a-421e-bf23-38d878371101-istiod-ca-cert\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.152356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.152344    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-credential-socket\") pod \"61d297af-035a-421e-bf23-38d878371101\" (UID: \"61d297af-035a-421e-bf23-38d878371101\") "
Oct 29 22:59:04.152514 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.152266    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d297af-035a-421e-bf23-38d878371101-istiod-ca-cert" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:04.152643 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.152622    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-credential-socket" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.153299 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.153282    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.153373 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.153366    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d297af-035a-421e-bf23-38d878371101-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.153434 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.153427    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.153492 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.153485    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.153553 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.153537    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.225013 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.224968    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-istio-token" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.232727 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.232649    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-kube-api-access-vxfnz" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "kube-api-access-vxfnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.237084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.234348    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d297af-035a-421e-bf23-38d878371101-istio-podinfo" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:04.241698 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.241184    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-envoy" pod "61d297af-035a-421e-bf23-38d878371101" (UID: "61d297af-035a-421e-bf23-38d878371101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.255477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.255362    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxfnz\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-kube-api-access-vxfnz\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.255477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.255397    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d297af-035a-421e-bf23-38d878371101-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.255477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.255411    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d297af-035a-421e-bf23-38d878371101-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.255477 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.255424    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d297af-035a-421e-bf23-38d878371101-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.325790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.325735    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:59:04.355963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.355933    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaabce08-691b-4828-8810-c426708e784e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaabce08-691b-4828-8810-c426708e784e-istio-podinfo\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356112 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356028    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaabce08-691b-4828-8810-c426708e784e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaabce08-691b-4828-8810-c426708e784e-istiod-ca-cert\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356112 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356072    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-kube-api-access-thnx5\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-kube-api-access-thnx5\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356158    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-istio-token\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-istio-token\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356304 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356248    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-socket\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356304 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356286    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-envoy\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356415 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356361    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-data\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356491    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-credential-socket\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.356575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356546    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-certs\") pod \"aaabce08-691b-4828-8810-c426708e784e\" (UID: \"aaabce08-691b-4828-8810-c426708e784e\") "
Oct 29 22:59:04.357265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.356997    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaabce08-691b-4828-8810-c426708e784e-istiod-ca-cert" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:04.357265 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.357219    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-certs" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.357594 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.357405    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-data" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.358042 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.357999    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-credential-socket" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.358363 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.358331    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.358363 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.358355    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.358637 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.358369    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaabce08-691b-4828-8810-c426708e784e-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.358637 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.358381    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.358938 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.358908    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-socket" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.359072 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.358934    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-istio-token" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.361289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.361244    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-envoy" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:04.392942 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.361410    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-kube-api-access-thnx5" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "kube-api-access-thnx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.392942 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.371399    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaabce08-691b-4828-8810-c426708e784e-istio-podinfo" pod "aaabce08-691b-4828-8810-c426708e784e" (UID: "aaabce08-691b-4828-8810-c426708e784e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:04.680548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.676811    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.680548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.676848    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaabce08-691b-4828-8810-c426708e784e-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.680548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.676862    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaabce08-691b-4828-8810-c426708e784e-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.680548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.676877    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thnx5\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-kube-api-access-thnx5\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.680548 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.676892    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaabce08-691b-4828-8810-c426708e784e-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.816663 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.816637    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:04.924791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.919270    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-xks-ca\") pod \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") "
Oct 29 22:59:04.924791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.919521    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-kube-api-access-gmc7b\" (UniqueName: \"kubernetes.io/projected/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-kube-api-access-gmc7b\") pod \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") "
Oct 29 22:59:04.924791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.919658    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-openshift-ca\") pod \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\" (UID: \"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c\") "
Oct 29 22:59:04.924791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.920803    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-openshift-ca" pod "5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" (UID: "5757d0ff-f91b-4b0f-bd12-bfca8bd6952c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:04.926033 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.925756    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-xks-ca" pod "5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" (UID: "5757d0ff-f91b-4b0f-bd12-bfca8bd6952c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:04.927904 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.927869    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-kube-api-access-gmc7b" pod "5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" (UID: "5757d0ff-f91b-4b0f-bd12-bfca8bd6952c"). InnerVolumeSpecName "kube-api-access-gmc7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.936816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.936561    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"
Oct 29 22:59:04.936816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.936639    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn" event={"ID":"aaabce08-691b-4828-8810-c426708e784e","Type":"ContainerDied","Data":"ab643cd0b812dab0376df64a062e52ae0e9c2785f768dee8b6e526fdb3590394"}
Oct 29 22:59:04.936816 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.936695    4827 scope.go:122] "RemoveContainer" containerID="2f8ca54a5e0481d05629c9b7710ef29ea315868b0c7afa0be1908e1b61a826d1"
Oct 29 22:59:04.940641 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.940615    4827 generic.go:356] "Generic (PLEG): container finished" podID="361dc423-b463-4373-ae2e-d4f6cf5e7a4c" containerID="1e0b7f7abeae5331009ab6a412ff4073ddf6a6b6e7ecc5a6ef2337f8d4511fe6" exitCode=0
Oct 29 22:59:04.941678 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.940718    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5" event={"ID":"361dc423-b463-4373-ae2e-d4f6cf5e7a4c","Type":"ContainerDied","Data":"1e0b7f7abeae5331009ab6a412ff4073ddf6a6b6e7ecc5a6ef2337f8d4511fe6"}
Oct 29 22:59:04.944439 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.944422    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"
Oct 29 22:59:04.944508 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.944470    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74" event={"ID":"5757d0ff-f91b-4b0f-bd12-bfca8bd6952c","Type":"ContainerDied","Data":"948b7ae8bf6c692118ec46d42ca7a12934369993a33b0baec8e4c67488d48ae6"}
Oct 29 22:59:04.944558 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.944508    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948b7ae8bf6c692118ec46d42ca7a12934369993a33b0baec8e4c67488d48ae6"
Oct 29 22:59:04.946436 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.946420    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"
Oct 29 22:59:04.947121 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:04.946354    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r" event={"ID":"61d297af-035a-421e-bf23-38d878371101","Type":"ContainerDied","Data":"a15697f266ade7a79fd1fe01ed929da52a31ba2d24647e262d8acba5bb661ecc"}
Oct 29 22:59:05.023422 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.023383    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:05.023422 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.023418    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmc7b\" (UniqueName: \"kubernetes.io/projected/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-kube-api-access-gmc7b\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:05.023671 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.023431    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:05.029800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.029779    4827 scope.go:122] "RemoveContainer" containerID="d814cbe1adbaffa35e08107787be07f3102568d36db71434c1e4af8967507da7"
Oct 29 22:59:05.031718 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.031700    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:59:05.124107 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.124070    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-plugins-config-volume\") pod \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") "
Oct 29 22:59:05.124334 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.124313    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-kube-api-access-vmlzr\" (UniqueName: \"kubernetes.io/projected/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-kube-api-access-vmlzr\") pod \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\" (UID: \"361dc423-b463-4373-ae2e-d4f6cf5e7a4c\") "
Oct 29 22:59:05.129420 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.129384    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-plugins-config-volume" pod "361dc423-b463-4373-ae2e-d4f6cf5e7a4c" (UID: "361dc423-b463-4373-ae2e-d4f6cf5e7a4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:05.131996 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.131933    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:05.133260 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.133235    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"]
Oct 29 22:59:05.137789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.137733    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-kube-api-access-vmlzr" pod "361dc423-b463-4373-ae2e-d4f6cf5e7a4c" (UID: "361dc423-b463-4373-ae2e-d4f6cf5e7a4c"). InnerVolumeSpecName "kube-api-access-vmlzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:05.141509 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.141470    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-9ng2r"]
Oct 29 22:59:05.147880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.147831    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"]
Oct 29 22:59:05.153345 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.153327    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-g9hvn"]
Oct 29 22:59:05.234062 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.233965    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmlzr\" (UniqueName: \"kubernetes.io/projected/361dc423-b463-4373-ae2e-d4f6cf5e7a4c-kube-api-access-vmlzr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:05.958793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.958749    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"
Oct 29 22:59:05.959191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.958853    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5" event={"ID":"361dc423-b463-4373-ae2e-d4f6cf5e7a4c","Type":"ContainerDied","Data":"cd203051ca75ba11867b4c5d73ca48c679a6edb8bcfe1fdea0293c71f5445027"}
Oct 29 22:59:05.959191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:05.958930    4827 scope.go:122] "RemoveContainer" containerID="1e0b7f7abeae5331009ab6a412ff4073ddf6a6b6e7ecc5a6ef2337f8d4511fe6"
Oct 29 22:59:06.107664 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:06.107617    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"]
Oct 29 22:59:06.129797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:06.129740    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-7765c7897b-56fl5"]
Oct 29 22:59:06.522981 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:06.522949    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361dc423-b463-4373-ae2e-d4f6cf5e7a4c" path="/var/lib/kubelet/pods/361dc423-b463-4373-ae2e-d4f6cf5e7a4c/volumes"
Oct 29 22:59:06.523574 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:06.523550    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d297af-035a-421e-bf23-38d878371101" path="/var/lib/kubelet/pods/61d297af-035a-421e-bf23-38d878371101/volumes"
Oct 29 22:59:06.524131 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:06.524116    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaabce08-691b-4828-8810-c426708e784e" path="/var/lib/kubelet/pods/aaabce08-691b-4828-8810-c426708e784e/volumes"
Oct 29 22:59:10.928930 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.928892    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"]
Oct 29 22:59:10.930057 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930038    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" containerName="revoke-keys"
Oct 29 22:59:10.930201 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930189    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" containerName="revoke-keys"
Oct 29 22:59:10.930302 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930293    4827 cpu_manager.go:414] "containerMap: removing container" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy"
Oct 29 22:59:10.930372 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930363    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy"
Oct 29 22:59:10.930470 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930461    4827 cpu_manager.go:414] "containerMap: removing container" podUID="361dc423-b463-4373-ae2e-d4f6cf5e7a4c" containerName="payload-processing"
Oct 29 22:59:10.930540 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930531    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361dc423-b463-4373-ae2e-d4f6cf5e7a4c" containerName="payload-processing"
Oct 29 22:59:10.930626 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930618    4827 cpu_manager.go:414] "containerMap: removing container" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy"
Oct 29 22:59:10.930692 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930684    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy"
Oct 29 22:59:10.930785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930762    4827 cpu_manager.go:414] "containerMap: removing container" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy"
Oct 29 22:59:10.930865 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.930853    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy"
Oct 29 22:59:10.931191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.931176    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy"
Oct 29 22:59:10.931298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.931289    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d297af-035a-421e-bf23-38d878371101" containerName="istio-proxy"
Oct 29 22:59:10.931383 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.931374    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" containerName="revoke-keys"
Oct 29 22:59:10.931458 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.931450    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="361dc423-b463-4373-ae2e-d4f6cf5e7a4c" containerName="payload-processing"
Oct 29 22:59:10.978559 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.978521    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"]
Oct 29 22:59:10.978782 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:10.978641    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.004242 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.004208    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-49dbb5-openshift-default-dockercfg-v22n2\""
Oct 29 22:59:11.043399 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.043367    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-data\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.043779 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.043740    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-token\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.043993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.043965    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-certs\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.043993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.044004    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-podinfo\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.044204 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.044041    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-credential-socket\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.044204 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.044093    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlcrt\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-kube-api-access-nlcrt\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.044204 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.044168    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-socket\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.044369 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.044247    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6d96c-910a-416e-aa2b-87bbf273de36-istiod-ca-cert\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.044451 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.044421    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-envoy\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.209589    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-token\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.252657    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-certs\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.252699    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-podinfo\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.252743    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-credential-socket\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.252827    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlcrt\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-kube-api-access-nlcrt\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.252939    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-socket\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.253070    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6d96c-910a-416e-aa2b-87bbf273de36-istiod-ca-cert\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.253374    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-envoy\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.253538    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-data\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.254223    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-data\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.254588    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-certs\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.260738    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-socket\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.261448    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-credential-socket\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.292857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.262819    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6d96c-910a-416e-aa2b-87bbf273de36-istiod-ca-cert\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.339960 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.332789    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-podinfo\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.347788 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.342411    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlcrt\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-kube-api-access-nlcrt\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.502871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.448621    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-token\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.509834 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.509527    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-envoy\") pod \"e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") " pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:11.685992 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:11.685693    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:12.269065 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:12.269035    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb6d96c_910a_416e_aa2b_87bbf273de36.slice/crio-d03f89c3a8b0971a6400b6bcdd6fdbd58bcfc056afed38d24db4b1a752257c3c WatchSource:0}: Error finding container d03f89c3a8b0971a6400b6bcdd6fdbd58bcfc056afed38d24db4b1a752257c3c: Status 404 returned error can't find the container with id d03f89c3a8b0971a6400b6bcdd6fdbd58bcfc056afed38d24db4b1a752257c3c
Oct 29 22:59:12.270038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.270009    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"]
Oct 29 22:59:12.279828 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.279733    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:12.279937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.279874    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:12.279937 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.279916    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:12.504202 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.504159    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"]
Oct 29 22:59:12.505628 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.505598    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaabce08-691b-4828-8810-c426708e784e" containerName="istio-proxy"
Oct 29 22:59:12.528968 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.528354    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:12.528968 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.528448    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"]
Oct 29 22:59:12.549221 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.549169    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f3092a4d\""
Oct 29 22:59:12.619786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.601280    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f3092a4d-dockercfg-6cd95\""
Oct 29 22:59:12.636834 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.627806    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28287833-a7b4-440c-b3e4-04c369bd615c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:12.636834 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.632617    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/28287833-a7b4-440c-b3e4-04c369bd615c-kube-api-access-xq9bg\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:12.766751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.762062    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"]
Oct 29 22:59:12.856528 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.821277    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/28287833-a7b4-440c-b3e4-04c369bd615c-kube-api-access-xq9bg\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:12.856528 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.821626    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28287833-a7b4-440c-b3e4-04c369bd615c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:12.928157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:12.881379    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28287833-a7b4-440c-b3e4-04c369bd615c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:13.085374 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.085339    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"]
Oct 29 22:59:13.085797 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.085776    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.140543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.140513    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpc6\" (UniqueName: \"kubernetes.io/projected/8c28df07-1331-4dc8-90dc-43d93c8ddae1-kube-api-access-4tpc6\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.140981 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.140956    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28df07-1331-4dc8-90dc-43d93c8ddae1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.205787 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.188443    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/28287833-a7b4-440c-b3e4-04c369bd615c-kube-api-access-xq9bg\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:13.270606 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.269381    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpc6\" (UniqueName: \"kubernetes.io/projected/8c28df07-1331-4dc8-90dc-43d93c8ddae1-kube-api-access-4tpc6\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.270606 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.269489    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28df07-1331-4dc8-90dc-43d93c8ddae1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.270606 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.270533    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28df07-1331-4dc8-90dc-43d93c8ddae1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.277494 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.277465    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" event={"ID":"0bb6d96c-910a-416e-aa2b-87bbf273de36","Type":"ContainerStarted","Data":"2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271"}
Oct 29 22:59:13.277711 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.277695    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" event={"ID":"0bb6d96c-910a-416e-aa2b-87bbf273de36","Type":"ContainerStarted","Data":"d03f89c3a8b0971a6400b6bcdd6fdbd58bcfc056afed38d24db4b1a752257c3c"}
Oct 29 22:59:13.294051 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.294005    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpc6\" (UniqueName: \"kubernetes.io/projected/8c28df07-1331-4dc8-90dc-43d93c8ddae1-kube-api-access-4tpc6\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.324375 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.324314    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podStartSLOduration=3.324295099 podStartE2EDuration="3.324295099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:13.322248974 +0000 UTC m=+988.945159995" watchObservedRunningTime="2026-10-29 22:59:13.324295099 +0000 UTC m=+988.947206122"
Oct 29 22:59:13.489844 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.483203    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:13.535725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.535475    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:13.692500 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.692131    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:13.699689 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.699653    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:14.073186 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:13.992761    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"]
Oct 29 22:59:14.073186 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:13.992880    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c28df07_1331_4dc8_90dc_43d93c8ddae1.slice/crio-1a1d315fc451ecf17b8ef198068a3f9c0ce206a9199320c2eeab4aecec16a8fa WatchSource:0}: Error finding container 1a1d315fc451ecf17b8ef198068a3f9c0ce206a9199320c2eeab4aecec16a8fa: Status 404 returned error can't find the container with id 1a1d315fc451ecf17b8ef198068a3f9c0ce206a9199320c2eeab4aecec16a8fa
Oct 29 22:59:14.073186 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.028167    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"]
Oct 29 22:59:14.324273 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.323442    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl" event={"ID":"8c28df07-1331-4dc8-90dc-43d93c8ddae1","Type":"ContainerStarted","Data":"7cb8b8e24118a98fdde5fce551d4d0239538f97091ab08bc7a571221b0115c92"}
Oct 29 22:59:14.324273 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.323510    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl" event={"ID":"8c28df07-1331-4dc8-90dc-43d93c8ddae1","Type":"ContainerStarted","Data":"1a1d315fc451ecf17b8ef198068a3f9c0ce206a9199320c2eeab4aecec16a8fa"}
Oct 29 22:59:14.324273 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.323875    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:14.332758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.331153    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd" event={"ID":"28287833-a7b4-440c-b3e4-04c369bd615c","Type":"ContainerStarted","Data":"0c94d7fdfc70da120d766dca91eaf8387b092a2b6aaf32f5c5353da089f661da"}
Oct 29 22:59:14.332758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.331214    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd" event={"ID":"28287833-a7b4-440c-b3e4-04c369bd615c","Type":"ContainerStarted","Data":"aed27c2d2523444a8a26e4e04456d43c6162a1472ee6400d6ae29b67b69629fb"}
Oct 29 22:59:14.332758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.331242    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:14.332758 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.331269    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:14.334906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.334721    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 22:59:14.368403 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.368347    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl" podStartSLOduration=2.368329881 podStartE2EDuration="2.368329881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:14.365963227 +0000 UTC m=+989.988874310" watchObservedRunningTime="2026-10-29 22:59:14.368329881 +0000 UTC m=+989.991240902"
Oct 29 22:59:14.464447 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.464375    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd" podStartSLOduration=2.464356528 podStartE2EDuration="2.464356528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:14.454635911 +0000 UTC m=+990.077547017" watchObservedRunningTime="2026-10-29 22:59:14.464356528 +0000 UTC m=+990.087267549"
Oct 29 22:59:14.688072 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.687999    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"]
Oct 29 22:59:14.774857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.773139    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.774857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.773545    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"]
Oct 29 22:59:14.789055 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.789031    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:14.789357 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.789323    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:59:14.851200 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.851161    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.851386 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.851331    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfsqz\" (UniqueName: \"kubernetes.io/projected/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-kube-api-access-hfsqz\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.851716 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.851695    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.953744 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.953664    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfsqz\" (UniqueName: \"kubernetes.io/projected/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-kube-api-access-hfsqz\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.954005 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.953989    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.954084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.954065    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.954751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.954726    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.956361 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.956331    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:14.977160 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:14.977126    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfsqz\" (UniqueName: \"kubernetes.io/projected/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-kube-api-access-hfsqz\") pod \"maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:15.091713 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.091672    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:15.338623 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.338592    4827 generic.go:356] "Generic (PLEG): container finished" podID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerID="b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44" exitCode=0
Oct 29 22:59:15.339191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.338670    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" event={"ID":"a79b06ae-bc79-45a0-acec-9462a314fba1","Type":"ContainerDied","Data":"b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44"}
Oct 29 22:59:15.339310 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.339291    4827 scope.go:122] "RemoveContainer" containerID="b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44"
Oct 29 22:59:15.340616 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.340597    4827 generic.go:356] "Generic (PLEG): container finished" podID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerID="475fd2bdd94f3d24ba1d52b3c50bb0312f38b15ad30df1884b58b42090e204e8" exitCode=0
Oct 29 22:59:15.340719 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.340674    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" event={"ID":"8b985efd-ab05-4960-ae31-13c61bf74b1b","Type":"ContainerDied","Data":"475fd2bdd94f3d24ba1d52b3c50bb0312f38b15ad30df1884b58b42090e204e8"}
Oct 29 22:59:15.341099 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.341069    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:15.341177 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.341151    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:15.341217 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.341198    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:15.341521 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.341506    4827 scope.go:122] "RemoveContainer" containerID="475fd2bdd94f3d24ba1d52b3c50bb0312f38b15ad30df1884b58b42090e204e8"
Oct 29 22:59:15.342922 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.342890    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:15.343027 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.342970    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:15.343092 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.343022    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:15.403488 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:15.403452    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6e2b2a_3a3a_4660_bae2_aaf5384b4243.slice/crio-d72fb4a3f1db6d48900cfe3a97aad5501906325940bc745bbe7b660cc219e7c8 WatchSource:0}: Error finding container d72fb4a3f1db6d48900cfe3a97aad5501906325940bc745bbe7b660cc219e7c8: Status 404 returned error can't find the container with id d72fb4a3f1db6d48900cfe3a97aad5501906325940bc745bbe7b660cc219e7c8
Oct 29 22:59:15.408877 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:15.408853    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"]
Oct 29 22:59:16.418800 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:16.418588    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" event={"ID":"a79b06ae-bc79-45a0-acec-9462a314fba1","Type":"ContainerStarted","Data":"f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a"}
Oct 29 22:59:16.448053 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:16.448000    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z" event={"ID":"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243","Type":"ContainerStarted","Data":"48dbab585e7ca4d2bb7c9c3b1f9b75f6e15fee4955b580c04f9ff31edc043d90"}
Oct 29 22:59:16.448379 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:16.448332    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z" event={"ID":"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243","Type":"ContainerStarted","Data":"d72fb4a3f1db6d48900cfe3a97aad5501906325940bc745bbe7b660cc219e7c8"}
Oct 29 22:59:16.770034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:16.747988    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:16.770034 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:16.748060    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:17.147030 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.146966    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:17.474989 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.474880    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:17.499220 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.491203    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:59:17.499497 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.499479    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" event={"ID":"8b985efd-ab05-4960-ae31-13c61bf74b1b","Type":"ContainerStarted","Data":"2c42d9deda369dfd59b9612ac3e0b1f84a052f57f1570d2cff1e62bffcd3d819"}
Oct 29 22:59:17.512792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.493575    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z" podStartSLOduration=3.493553925 podStartE2EDuration="3.493553925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:16.895882508 +0000 UTC m=+992.518793529" watchObservedRunningTime="2026-10-29 22:59:17.493553925 +0000 UTC m=+993.116464946"
Oct 29 22:59:17.597962 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.597903    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:17.682493 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.680320    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:59:17.695089 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.695044    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:17.699220 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.699189    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:17.699505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.699460    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:17.732841 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.731249    4827 generic.go:356] "Generic (PLEG): container finished" podID="ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" containerID="48dbab585e7ca4d2bb7c9c3b1f9b75f6e15fee4955b580c04f9ff31edc043d90" exitCode=0
Oct 29 22:59:17.734070 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.734037    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z" event={"ID":"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243","Type":"ContainerDied","Data":"48dbab585e7ca4d2bb7c9c3b1f9b75f6e15fee4955b580c04f9ff31edc043d90"}
Oct 29 22:59:17.803105 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.803058    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:17.866590 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.866552    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"]
Oct 29 22:59:17.894625 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.893007    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:59:17.894625 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.893736    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 22:59:17.921621 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.921595    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a37a29-gntd2"]
Oct 29 22:59:17.928929 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.893837    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 22:59:17.962855 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:17.961846    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:59:18.559461 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:18.559421    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab71a29-b7b6-4854-ba77-4c1be33ec750" path="/var/lib/kubelet/pods/4ab71a29-b7b6-4854-ba77-4c1be33ec750/volumes"
Oct 29 22:59:18.710023 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:18.709987    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:18.710321 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:18.710290    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:18.893805 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:18.882905    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 22:59:18.906880 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:18.906833    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 22:59:19.710118 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.707142    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:19.710118 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.707199    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:19.730836 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.730653    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:19.829289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.829250    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z" event={"ID":"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243","Type":"ContainerDied","Data":"d72fb4a3f1db6d48900cfe3a97aad5501906325940bc745bbe7b660cc219e7c8"}
Oct 29 22:59:19.829537 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.829516    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"
Oct 29 22:59:19.829893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.829578    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d72fb4a3f1db6d48900cfe3a97aad5501906325940bc745bbe7b660cc219e7c8"
Oct 29 22:59:19.845629 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.845598    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-openshift-ca\") pod \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") "
Oct 29 22:59:19.845834 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.845728    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-kube-api-access-hfsqz\" (UniqueName: \"kubernetes.io/projected/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-kube-api-access-hfsqz\") pod \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") "
Oct 29 22:59:19.845992 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.845969    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-openshift-ca" pod "ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" (UID: "ec6e2b2a-3a3a-4660-bae2-aaf5384b4243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:19.846064 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.846048    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-xks-ca\") pod \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\" (UID: \"ec6e2b2a-3a3a-4660-bae2-aaf5384b4243\") "
Oct 29 22:59:19.846854 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.846831    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:19.849703 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.848390    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-xks-ca" pod "ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" (UID: "ec6e2b2a-3a3a-4660-bae2-aaf5384b4243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:19.850857 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.850113    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-kube-api-access-hfsqz" pod "ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" (UID: "ec6e2b2a-3a3a-4660-bae2-aaf5384b4243"). InnerVolumeSpecName "kube-api-access-hfsqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:19.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.897052    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 22:59:19.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.897104    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 22:59:19.948490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.948453    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfsqz\" (UniqueName: \"kubernetes.io/projected/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-kube-api-access-hfsqz\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:19.948490 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:19.948486    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:20.737178 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:20.737047    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:20.757346 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:20.757250    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:20.892831 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:20.892798    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 22:59:20.893074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:20.893045    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 22:59:21.031963 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.029332    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:21.055756 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.052537    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:21.157957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.157902    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:59:21.251307 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.251268    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:21.709792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.709185    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:21.709792 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.709259    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:21.896789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.881257    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 22:59:21.896789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:21.881319    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 22:59:22.702697 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:22.702662    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:22.702998 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:22.702971    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:22.891790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:22.880112    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 22:59:22.891790 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:22.880183    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 22:59:23.704793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:23.704306    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:59:23.704793 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:23.704384    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 22:59:23.915789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:23.915727    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 22:59:23.915993 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:23.915838    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 22:59:24.702941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:24.702914    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:59:24.704459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:24.704431    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 22:59:24.885279 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:24.885247    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:59:24.886575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:24.886553    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 22:59:24.996083 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:24.996001    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"]
Oct 29 22:59:24.996672 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:24.996641    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy" containerID="cri-o://51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9" gracePeriod=30
Oct 29 22:59:25.416913 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:25.416870    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"]
Oct 29 22:59:25.513726 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:25.513690    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 22:59:25.513941 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:25.513794    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 22:59:26.674223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.605430    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"]
Oct 29 22:59:26.674223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.630423    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" containerName="revoke-keys"
Oct 29 22:59:26.674223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.630450    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" containerName="revoke-keys"
Oct 29 22:59:26.674223 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.630953    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" containerName="revoke-keys"
Oct 29 22:59:26.729141 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.718930    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:26.747786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.734011    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"]
Oct 29 22:59:26.747786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.734146    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.784791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.783052    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 22:59:26.784791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.783100    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:26.850976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.849177    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.850976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.849324    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwkl\" (UniqueName: \"kubernetes.io/projected/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-kube-api-access-nnwkl\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.850976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.849552    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.976906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.954266    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.976906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.954374    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwkl\" (UniqueName: \"kubernetes.io/projected/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-kube-api-access-nnwkl\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.976906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.954584    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.976906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.956379    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:26.976906 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.976823    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:27.008785 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:26.998422    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwkl\" (UniqueName: \"kubernetes.io/projected/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-kube-api-access-nnwkl\") pod \"maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:27.077289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:27.077253    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:27.137791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:27.125466    4827 patch_prober.go:29] interesting pod/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:59:27.137791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:27.125525    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:59:27.470658 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:27.470576    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"]
Oct 29 22:59:27.496402 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:27.496368    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7274b6_9bbb_4cea_95c4_c6d8e7a7b75d.slice/crio-704dc4093129c01a573ca067ac216128e34c9ddab447c6343e26717e286e9d27 WatchSource:0}: Error finding container 704dc4093129c01a573ca067ac216128e34c9ddab447c6343e26717e286e9d27: Status 404 returned error can't find the container with id 704dc4093129c01a573ca067ac216128e34c9ddab447c6343e26717e286e9d27
Oct 29 22:59:27.622789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:27.619693    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd" event={"ID":"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d","Type":"ContainerStarted","Data":"704dc4093129c01a573ca067ac216128e34c9ddab447c6343e26717e286e9d27"}
Oct 29 22:59:28.630396 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:28.630361    4827 generic.go:356] "Generic (PLEG): container finished" podID="ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" containerID="cea57decfdfeaa6bc924231e87d686d685d770e7760152f93bef44a59d2b43e1" exitCode=0
Oct 29 22:59:28.630893 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:28.630446    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd" event={"ID":"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d","Type":"ContainerDied","Data":"cea57decfdfeaa6bc924231e87d686d685d770e7760152f93bef44a59d2b43e1"}
Oct 29 22:59:29.833304 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.833238    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:29.967241 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.967166    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-xks-ca\") pod \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") "
Oct 29 22:59:29.967384 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.967280    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-kube-api-access-nnwkl\" (UniqueName: \"kubernetes.io/projected/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-kube-api-access-nnwkl\") pod \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") "
Oct 29 22:59:29.967384 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.967382    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-openshift-ca\") pod \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\" (UID: \"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d\") "
Oct 29 22:59:29.967783 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.967745    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-openshift-ca" pod "ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" (UID: "ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:29.968100 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.968080    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:29.969342 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.969318    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-xks-ca" pod "ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" (UID: "ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:29.969807 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:29.969787    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-kube-api-access-nnwkl" pod "ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" (UID: "ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d"). InnerVolumeSpecName "kube-api-access-nnwkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:30.069539 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.069510    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwkl\" (UniqueName: \"kubernetes.io/projected/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-kube-api-access-nnwkl\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.069539 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.069539    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.251451 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.251422    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:59:30.374485 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374446    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-data\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374507    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-credential-socket\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374539    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-podinfo\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374597    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istiod-ca-cert\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374676 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374671    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-socket\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374707    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-token\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-token\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374742    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-certs\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374801    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-data" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:30.374959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374813    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-kube-api-access-8ws6q\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-kube-api-access-8ws6q\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.374959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.374872    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-credential-socket" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:30.375319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.375128    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-socket" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:30.375319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.375165    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istiod-ca-cert" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:30.375319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.375231    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-certs" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:30.375319 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.375313    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-envoy\") pod \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\" (UID: \"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b\") "
Oct 29 22:59:30.376267 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.376239    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.376408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.376271    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.376408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.376285    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.376408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.376299    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.376408 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.376312    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.377220 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.377197    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-podinfo" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:30.377298 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.377258    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-token" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:30.377357 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.377343    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-kube-api-access-8ws6q" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "kube-api-access-8ws6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:30.377759 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.377744    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-envoy" pod "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" (UID: "66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:30.440964 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.440931    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"]
Oct 29 22:59:30.441832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.441798    4827 cpu_manager.go:414] "containerMap: removing container" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy"
Oct 29 22:59:30.441832 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.441832    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy"
Oct 29 22:59:30.442038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.441872    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" containerName="revoke-keys"
Oct 29 22:59:30.442038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.441879    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" containerName="revoke-keys"
Oct 29 22:59:30.442142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.442102    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy"
Oct 29 22:59:30.442142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.442121    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy"
Oct 29 22:59:30.442142 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.442134    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" containerName="revoke-keys"
Oct 29 22:59:30.445297 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.445278    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.448555 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.448518    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm" podUID="93b04dd1-62ce-4858-b8e8-2ae0648b11af" containerName="payload-pre-processing" containerID="cri-o://2ee6ff975a6f21228d1511b837e7cda956fabbfea807b1a0c79d329343049550" gracePeriod=25
Oct 29 22:59:30.459987 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.459956    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3092a4d-serving-cert\""
Oct 29 22:59:30.468065 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.468040    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"]
Oct 29 22:59:30.477996 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.477976    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.477996 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.477997    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.478165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.478007    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ws6q\" (UniqueName: \"kubernetes.io/projected/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-kube-api-access-8ws6q\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.478165 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.478018    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.580053 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.580027    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-maas-api-tls\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.580291 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.580248    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-metrics-certs\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.580409 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.580334    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9rb\" (UniqueName: \"kubernetes.io/projected/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-kube-api-access-5p9rb\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.580598 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.580578    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-openshift-service-ca\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.656327 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.656290    4827 generic.go:356] "Generic (PLEG): container finished" podID="93b04dd1-62ce-4858-b8e8-2ae0648b11af" containerID="2ee6ff975a6f21228d1511b837e7cda956fabbfea807b1a0c79d329343049550" exitCode=0
Oct 29 22:59:30.656543 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.656459    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm" event={"ID":"93b04dd1-62ce-4858-b8e8-2ae0648b11af","Type":"ContainerDied","Data":"2ee6ff975a6f21228d1511b837e7cda956fabbfea807b1a0c79d329343049550"}
Oct 29 22:59:30.659306 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.659235    4827 generic.go:356] "Generic (PLEG): container finished" podID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerID="51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9" exitCode=0
Oct 29 22:59:30.659545 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.659394    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" event={"ID":"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b","Type":"ContainerDied","Data":"51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9"}
Oct 29 22:59:30.659545 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.659485    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng" event={"ID":"66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b","Type":"ContainerDied","Data":"5535d94cf20daa3dd2cc0d09f2ba81e4147f03b3a49a394eabf946839ef63d17"}
Oct 29 22:59:30.659700 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.659547    4827 scope.go:122] "RemoveContainer" containerID="51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9"
Oct 29 22:59:30.660462 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.660438    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"
Oct 29 22:59:30.664976 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.664938    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd" event={"ID":"ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d","Type":"ContainerDied","Data":"704dc4093129c01a573ca067ac216128e34c9ddab447c6343e26717e286e9d27"}
Oct 29 22:59:30.665074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.664987    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="704dc4093129c01a573ca067ac216128e34c9ddab447c6343e26717e286e9d27"
Oct 29 22:59:30.665074 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.665049    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"
Oct 29 22:59:30.748809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.725902    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-maas-api-tls\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.748809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.726043    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-metrics-certs\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.748809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.726086    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9rb\" (UniqueName: \"kubernetes.io/projected/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-kube-api-access-5p9rb\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.748809 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.726465    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-openshift-service-ca\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.778725 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.778695    4827 scope.go:122] "RemoveContainer" containerID="f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3"
Oct 29 22:59:30.779310 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.779285    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-maas-api-tls\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.788810 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.727697    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-openshift-service-ca\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.796919 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.796882    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"]
Oct 29 22:59:30.902325 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.892903    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a37a29-openshift-default-86dbb8dbc-pwpng"]
Oct 29 22:59:30.905252 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.905161    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9rb\" (UniqueName: \"kubernetes.io/projected/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-kube-api-access-5p9rb\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.959781 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.959730    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-metrics-certs\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:30.976316 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.976191    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:59:30.979399 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.979378    4827 scope.go:122] "RemoveContainer" containerID="51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9"
Oct 29 22:59:30.979707 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:59:30.979688    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9\": container with ID starting with 51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9 not found: ID does not exist" containerID="51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9"
Oct 29 22:59:30.979840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.979719    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9"} err="failed to get container status \"51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9\": rpc error: code = NotFound desc = could not find container \"51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9\": container with ID starting with 51daab42af3757623c4b022c789eae71a8d820d7f9fd175af736250a3d7ef0d9 not found: ID does not exist"
Oct 29 22:59:30.979840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.979742    4827 scope.go:122] "RemoveContainer" containerID="f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3"
Oct 29 22:59:30.980123 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:59:30.980103    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3\": container with ID starting with f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3 not found: ID does not exist" containerID="f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3"
Oct 29 22:59:30.980182 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:30.980128    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3"} err="failed to get container status \"f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3\": rpc error: code = NotFound desc = could not find container \"f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3\": container with ID starting with f1bbcaf3ea481cee79f084ac34041586d9c6d2a716c1e2557ff03afc5082f5e3 not found: ID does not exist"
Oct 29 22:59:31.055802 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.055689    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b04dd1-62ce-4858-b8e8-2ae0648b11af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b04dd1-62ce-4858-b8e8-2ae0648b11af-plugins-config-volume\") pod \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") "
Oct 29 22:59:31.055989 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.055975    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b04dd1-62ce-4858-b8e8-2ae0648b11af-kube-api-access-pthwj\" (UniqueName: \"kubernetes.io/projected/93b04dd1-62ce-4858-b8e8-2ae0648b11af-kube-api-access-pthwj\") pod \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\" (UID: \"93b04dd1-62ce-4858-b8e8-2ae0648b11af\") "
Oct 29 22:59:31.056106 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.056077    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b04dd1-62ce-4858-b8e8-2ae0648b11af-plugins-config-volume" pod "93b04dd1-62ce-4858-b8e8-2ae0648b11af" (UID: "93b04dd1-62ce-4858-b8e8-2ae0648b11af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:31.056722 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.056699    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b04dd1-62ce-4858-b8e8-2ae0648b11af-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:31.058435 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.058410    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b04dd1-62ce-4858-b8e8-2ae0648b11af-kube-api-access-pthwj" pod "93b04dd1-62ce-4858-b8e8-2ae0648b11af" (UID: "93b04dd1-62ce-4858-b8e8-2ae0648b11af"). InnerVolumeSpecName "kube-api-access-pthwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:31.066861 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.066596    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:31.159208 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.159174    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthwj\" (UniqueName: \"kubernetes.io/projected/93b04dd1-62ce-4858-b8e8-2ae0648b11af-kube-api-access-pthwj\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:31.390581 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:31.390549    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64b6dcf_a9a6_4ba5_9d2d_6907a0c0ebbf.slice/crio-5739a351ef0755120214a28ba48747d2b89c38dda49616b634d9a81616a9b432 WatchSource:0}: Error finding container 5739a351ef0755120214a28ba48747d2b89c38dda49616b634d9a81616a9b432: Status 404 returned error can't find the container with id 5739a351ef0755120214a28ba48747d2b89c38dda49616b634d9a81616a9b432
Oct 29 22:59:31.391623 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.391592    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"]
Oct 29 22:59:31.736431 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.736289    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6" event={"ID":"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf","Type":"ContainerStarted","Data":"5739a351ef0755120214a28ba48747d2b89c38dda49616b634d9a81616a9b432"}
Oct 29 22:59:31.793639 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.790937    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm" event={"ID":"93b04dd1-62ce-4858-b8e8-2ae0648b11af","Type":"ContainerDied","Data":"528a87e0b81c764737c69eb3b2df213ad44e9f6d7bd4ec4e303fd7a4243ccfc7"}
Oct 29 22:59:31.795118 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.795093    4827 scope.go:122] "RemoveContainer" containerID="2ee6ff975a6f21228d1511b837e7cda956fabbfea807b1a0c79d329343049550"
Oct 29 22:59:31.795728 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.794483    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"
Oct 29 22:59:31.878120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.878094    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"]
Oct 29 22:59:31.884240 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:31.884218    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd525f-65b94fc87f-vsxjm"]
Oct 29 22:59:32.526417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:32.526385    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" path="/var/lib/kubelet/pods/66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b/volumes"
Oct 29 22:59:32.527173 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:32.527150    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b04dd1-62ce-4858-b8e8-2ae0648b11af" path="/var/lib/kubelet/pods/93b04dd1-62ce-4858-b8e8-2ae0648b11af/volumes"
Oct 29 22:59:32.812285 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:32.812196    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6" event={"ID":"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf","Type":"ContainerStarted","Data":"7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135"}
Oct 29 22:59:32.812460 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:32.812331    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:32.834674 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:59:32.834644    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 22:59:32.856499 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:32.856439    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6" podStartSLOduration=2.444485913 podStartE2EDuration="2.856425061s" totalImagesPullingTime="411.939148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:32.853322364 +0000 UTC m=+1008.476233385" watchObservedRunningTime="2026-10-29 22:59:32.856425061 +0000 UTC m=+1008.479336118"
Oct 29 22:59:36.218693 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:36.218658    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"]
Oct 29 22:59:36.253943 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:36.253914    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"]
Oct 29 22:59:38.648011 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.647978    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"]
Oct 29 22:59:38.649120 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.649096    4827 cpu_manager.go:414] "containerMap: removing container" podUID="93b04dd1-62ce-4858-b8e8-2ae0648b11af" containerName="payload-pre-processing"
Oct 29 22:59:38.649236 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.649123    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b04dd1-62ce-4858-b8e8-2ae0648b11af" containerName="payload-pre-processing"
Oct 29 22:59:38.649236 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.649193    4827 cpu_manager.go:414] "containerMap: removing container" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy"
Oct 29 22:59:38.649236 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.649202    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fbdc0a-f24a-4be2-a8bf-2152d3db1a5b" containerName="istio-proxy"
Oct 29 22:59:38.649401 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.649389    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b04dd1-62ce-4858-b8e8-2ae0648b11af" containerName="payload-pre-processing"
Oct 29 22:59:38.654977 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.654955    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.682063 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.682028    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d3e268-openshift-default-dockercfg-8tdq4\""
Oct 29 22:59:38.686295 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.686265    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"]
Oct 29 22:59:38.744203 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744162    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-envoy\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744441 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744413    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istiod-ca-cert\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744457    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-podinfo\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744587 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744544    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-token\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744687 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744601    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-socket\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744688    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-data\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744752 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744712    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-certs\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744747    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-credential-socket\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.744847 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.744816    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvt6\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-kube-api-access-btvt6\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.829294 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.829267    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.846610    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istiod-ca-cert\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.846678    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-podinfo\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.846751    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-token\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.846922    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-socket\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.847072    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-data\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.847105    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-certs\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.847149    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-credential-socket\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.847175    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvt6\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-kube-api-access-btvt6\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.847908 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.847283    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-envoy\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.849210 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.849094    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-socket\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.850048 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.850027    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istiod-ca-cert\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.861715 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.853059    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-certs\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.866238 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.862308    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-credential-socket\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.866505 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.863577    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-data\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.890157 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.890060    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-envoy\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:38.989324 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:38.989246    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvt6\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-kube-api-access-btvt6\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:39.116786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:39.083018    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-token\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:39.116786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:39.083536    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-podinfo\") pod \"e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") " pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:39.307839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:39.305806    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:39.756417 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:39.756370    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"]
Oct 29 22:59:39.759390 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:39.759357    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fadd7a_083a_4b36_8d31_6d89fd4e9739.slice/crio-e15aeb02ebeb8c1f82072aed67c18c7166d71ea01bde272a69c40a86081defd6 WatchSource:0}: Error finding container e15aeb02ebeb8c1f82072aed67c18c7166d71ea01bde272a69c40a86081defd6: Status 404 returned error can't find the container with id e15aeb02ebeb8c1f82072aed67c18c7166d71ea01bde272a69c40a86081defd6
Oct 29 22:59:39.762094 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:39.762062    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:39.762190 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:39.762137    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:39.762190 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:39.762165    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:40.125957 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:40.125921    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" event={"ID":"32fadd7a-083a-4b36-8d31-6d89fd4e9739","Type":"ContainerStarted","Data":"978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb"}
Oct 29 22:59:40.126136 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:40.125984    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" event={"ID":"32fadd7a-083a-4b36-8d31-6d89fd4e9739","Type":"ContainerStarted","Data":"e15aeb02ebeb8c1f82072aed67c18c7166d71ea01bde272a69c40a86081defd6"}
Oct 29 22:59:40.176437 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:40.176387    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podStartSLOduration=2.176373069 podStartE2EDuration="2.176373069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:40.170482492 +0000 UTC m=+1015.793393530" watchObservedRunningTime="2026-10-29 22:59:40.176373069 +0000 UTC m=+1015.799284088"
Oct 29 22:59:40.319791 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:40.317905    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:40.379839 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:40.379108    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:41.136119 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.136089    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:41.137125 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.137109    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 22:59:41.219056 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.219019    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx" podUID="8c182a50-c02c-4cb9-ada5-181183fd0118" containerName="payload-processing" containerID="cri-o://11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704" gracePeriod=25
Oct 29 22:59:41.255188 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.255151    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7" podUID="285da0f6-1f6d-4541-a5d7-d09d2297a793" containerName="payload-pre-processing" containerID="cri-o://9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8" gracePeriod=25
Oct 29 22:59:41.567704 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.567672    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:59:41.631087 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.631058    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:59:41.688927 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.688833    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285da0f6-1f6d-4541-a5d7-d09d2297a793-kube-api-access-bsxlb\" (UniqueName: \"kubernetes.io/projected/285da0f6-1f6d-4541-a5d7-d09d2297a793-kube-api-access-bsxlb\") pod \"285da0f6-1f6d-4541-a5d7-d09d2297a793\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") "
Oct 29 22:59:41.689075 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.688967    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285da0f6-1f6d-4541-a5d7-d09d2297a793-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285da0f6-1f6d-4541-a5d7-d09d2297a793-plugins-config-volume\") pod \"285da0f6-1f6d-4541-a5d7-d09d2297a793\" (UID: \"285da0f6-1f6d-4541-a5d7-d09d2297a793\") "
Oct 29 22:59:41.689205 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.689186    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c182a50-c02c-4cb9-ada5-181183fd0118-kube-api-access-c6spm\" (UniqueName: \"kubernetes.io/projected/8c182a50-c02c-4cb9-ada5-181183fd0118-kube-api-access-c6spm\") pod \"8c182a50-c02c-4cb9-ada5-181183fd0118\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") "
Oct 29 22:59:41.689361 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.689337    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c182a50-c02c-4cb9-ada5-181183fd0118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c182a50-c02c-4cb9-ada5-181183fd0118-plugins-config-volume\") pod \"8c182a50-c02c-4cb9-ada5-181183fd0118\" (UID: \"8c182a50-c02c-4cb9-ada5-181183fd0118\") "
Oct 29 22:59:41.690589 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.689607    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285da0f6-1f6d-4541-a5d7-d09d2297a793-plugins-config-volume" pod "285da0f6-1f6d-4541-a5d7-d09d2297a793" (UID: "285da0f6-1f6d-4541-a5d7-d09d2297a793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:41.690589 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.690044    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c182a50-c02c-4cb9-ada5-181183fd0118-plugins-config-volume" pod "8c182a50-c02c-4cb9-ada5-181183fd0118" (UID: "8c182a50-c02c-4cb9-ada5-181183fd0118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:41.690589 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.690381    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c182a50-c02c-4cb9-ada5-181183fd0118-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:41.690589 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.690403    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285da0f6-1f6d-4541-a5d7-d09d2297a793-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:41.691699 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.691673    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c182a50-c02c-4cb9-ada5-181183fd0118-kube-api-access-c6spm" pod "8c182a50-c02c-4cb9-ada5-181183fd0118" (UID: "8c182a50-c02c-4cb9-ada5-181183fd0118"). InnerVolumeSpecName "kube-api-access-c6spm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:41.691853 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.691830    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285da0f6-1f6d-4541-a5d7-d09d2297a793-kube-api-access-bsxlb" pod "285da0f6-1f6d-4541-a5d7-d09d2297a793" (UID: "285da0f6-1f6d-4541-a5d7-d09d2297a793"). InnerVolumeSpecName "kube-api-access-bsxlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:41.792575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.792538    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6spm\" (UniqueName: \"kubernetes.io/projected/8c182a50-c02c-4cb9-ada5-181183fd0118-kube-api-access-c6spm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:41.792575 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.792571    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxlb\" (UniqueName: \"kubernetes.io/projected/285da0f6-1f6d-4541-a5d7-d09d2297a793-kube-api-access-bsxlb\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 22:59:41.829210 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.829179    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"]
Oct 29 22:59:41.830038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.829998    4827 cpu_manager.go:414] "containerMap: removing container" podUID="285da0f6-1f6d-4541-a5d7-d09d2297a793" containerName="payload-pre-processing"
Oct 29 22:59:41.830038 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.830022    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285da0f6-1f6d-4541-a5d7-d09d2297a793" containerName="payload-pre-processing"
Oct 29 22:59:41.830245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.830048    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8c182a50-c02c-4cb9-ada5-181183fd0118" containerName="payload-processing"
Oct 29 22:59:41.830245 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.830056    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c182a50-c02c-4cb9-ada5-181183fd0118" containerName="payload-processing"
Oct 29 22:59:41.830357 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.830293    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="285da0f6-1f6d-4541-a5d7-d09d2297a793" containerName="payload-pre-processing"
Oct 29 22:59:41.830357 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.830309    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c182a50-c02c-4cb9-ada5-181183fd0118" containerName="payload-processing"
Oct 29 22:59:41.834511 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.834490    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.854084 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.854047    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-13aa41-openshift-default-dockercfg-2zxdv\""
Oct 29 22:59:41.860414 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.860385    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"]
Oct 29 22:59:41.894310 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894280    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.894459 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894353    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztng\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-kube-api-access-lztng\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.894524 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894485    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.894623 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894603    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469b8d53-8386-4f8b-942b-d0fedb13eb95-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.894691 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894650    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.894751 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894687    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.894842 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894828    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.894955 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894940    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:41.895011 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:41.894990    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.031951 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.031844    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469b8d53-8386-4f8b-942b-d0fedb13eb95-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.032251 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.032232    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.038881 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.032412    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.039687 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.039663    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.048891 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.048864    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.049178 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.049159    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.049446 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.049414    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.097116 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.097081    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztng\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-kube-api-access-lztng\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.097388 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.097371    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.097690 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.062319    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.097837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.062598    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.097959 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.065412    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469b8d53-8386-4f8b-942b-d0fedb13eb95-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.098051 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.067029    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.098140 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.050262    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.098503 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.061976    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.102973 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.102919    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.134863 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.122792    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.148799 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.136617    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztng\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-kube-api-access-lztng\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.162798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.160244    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:42.397789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.379720    4827 generic.go:356] "Generic (PLEG): container finished" podID="285da0f6-1f6d-4541-a5d7-d09d2297a793" containerID="9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8" exitCode=0
Oct 29 22:59:42.397789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.382851    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"
Oct 29 22:59:42.397789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.394209    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7" event={"ID":"285da0f6-1f6d-4541-a5d7-d09d2297a793","Type":"ContainerDied","Data":"9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8"}
Oct 29 22:59:42.397789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.394380    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7" event={"ID":"285da0f6-1f6d-4541-a5d7-d09d2297a793","Type":"ContainerDied","Data":"98aa7a055491e9c79790764f2a4e38e5291d0efcf3457c79a1cd724fd5aae1c2"}
Oct 29 22:59:42.397789 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.394696    4827 scope.go:122] "RemoveContainer" containerID="9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8"
Oct 29 22:59:42.624840 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.620606    4827 generic.go:356] "Generic (PLEG): container finished" podID="8c182a50-c02c-4cb9-ada5-181183fd0118" containerID="11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704" exitCode=0
Oct 29 22:59:42.657798 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.639848    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"
Oct 29 22:59:42.735862 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.735828    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx" event={"ID":"8c182a50-c02c-4cb9-ada5-181183fd0118","Type":"ContainerDied","Data":"11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704"}
Oct 29 22:59:42.736096 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.736077    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx" event={"ID":"8c182a50-c02c-4cb9-ada5-181183fd0118","Type":"ContainerDied","Data":"116b8b30fcbe5ce9368eabc953fa227f79ec1a081d4aebd584d307e28e31c421"}
Oct 29 22:59:42.743356 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.743125    4827 scope.go:122] "RemoveContainer" containerID="9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8"
Oct 29 22:59:42.744878 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:59:42.744853    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8\": container with ID starting with 9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8 not found: ID does not exist" containerID="9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8"
Oct 29 22:59:42.745021 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.744993    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8"} err="failed to get container status \"9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8\": rpc error: code = NotFound desc = could not find container \"9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8\": container with ID starting with 9933f7142c5e237e2fe178f7df75c480ea253079ca9237c8d039697368c4b7e8 not found: ID does not exist"
Oct 29 22:59:42.745138 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.745127    4827 scope.go:122] "RemoveContainer" containerID="11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704"
Oct 29 22:59:42.796578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.777069    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"]
Oct 29 22:59:42.796578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.786227    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a0dbf5-64c586f5b7-cwvx7"]
Oct 29 22:59:42.796578 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.793059    4827 scope.go:122] "RemoveContainer" containerID="11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704"
Oct 29 22:59:42.929285 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:42.929200    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"]
Oct 29 22:59:43.090921 ip-10-0-134-205 kubenswrapper[4827]: E1029 22:59:43.090867    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704\": container with ID starting with 11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704 not found: ID does not exist" containerID="11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704"
Oct 29 22:59:43.091303 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.091275    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704"} err="failed to get container status \"11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704\": rpc error: code = NotFound desc = could not find container \"11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704\": container with ID starting with 11803810d4d60f4f423b7ffef88552edccc2f0f108bd8786a9f1d4d5a4c32704 not found: ID does not exist"
Oct 29 22:59:43.304419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.304391    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a0dbf5-57cb95ddd4-hfxjx"]
Oct 29 22:59:43.308865 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:43.308630    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469b8d53_8386_4f8b_942b_d0fedb13eb95.slice/crio-139a0f02b5832499821b8d85df967a1ea3f6a412b34f58f2ec49a6a327c9e595 WatchSource:0}: Error finding container 139a0f02b5832499821b8d85df967a1ea3f6a412b34f58f2ec49a6a327c9e595: Status 404 returned error can't find the container with id 139a0f02b5832499821b8d85df967a1ea3f6a412b34f58f2ec49a6a327c9e595
Oct 29 22:59:43.312166 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.312123    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:43.312289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.312218    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:43.312289 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.312274    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 22:59:43.315527 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.315483    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"]
Oct 29 22:59:43.632848 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.632813    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" event={"ID":"469b8d53-8386-4f8b-942b-d0fedb13eb95","Type":"ContainerStarted","Data":"889125e7031170fddd986093a37a3dbac36ed904375dd0fe4805ae8545935eb1"}
Oct 29 22:59:43.633059 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.632878    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" event={"ID":"469b8d53-8386-4f8b-942b-d0fedb13eb95","Type":"ContainerStarted","Data":"139a0f02b5832499821b8d85df967a1ea3f6a412b34f58f2ec49a6a327c9e595"}
Oct 29 22:59:43.690416 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:43.690367    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podStartSLOduration=2.690353001 podStartE2EDuration="2.690353001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:43.677401031 +0000 UTC m=+1019.300312076" watchObservedRunningTime="2026-10-29 22:59:43.690353001 +0000 UTC m=+1019.313264021"
Oct 29 22:59:44.165357 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.165324    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:44.172151 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.172126    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:44.490568 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.490492    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285da0f6-1f6d-4541-a5d7-d09d2297a793" path="/var/lib/kubelet/pods/285da0f6-1f6d-4541-a5d7-d09d2297a793/volumes"
Oct 29 22:59:44.491060 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.491045    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c182a50-c02c-4cb9-ada5-181183fd0118" path="/var/lib/kubelet/pods/8c182a50-c02c-4cb9-ada5-181183fd0118/volumes"
Oct 29 22:59:44.676706 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.676677    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:44.691464 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.682202    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 22:59:44.847070 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.846996    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"]
Oct 29 22:59:44.853354 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.853328    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:44.874305 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.874276    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d3e268-dockercfg-xd2dg\""
Oct 29 22:59:44.875320 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.875293    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d3e268\""
Oct 29 22:59:44.878878 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.878224    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"]
Oct 29 22:59:44.991796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.985580    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2sn\" (UniqueName: \"kubernetes.io/projected/f34adf7f-7dff-4502-ba91-099f0bf0be62-kube-api-access-ht2sn\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:44.991796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:44.986073    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34adf7f-7dff-4502-ba91-099f0bf0be62-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:45.112837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.092749    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34adf7f-7dff-4502-ba91-099f0bf0be62-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:45.112837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.092841    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2sn\" (UniqueName: \"kubernetes.io/projected/f34adf7f-7dff-4502-ba91-099f0bf0be62-kube-api-access-ht2sn\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:45.112837 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.098362    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34adf7f-7dff-4502-ba91-099f0bf0be62-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:45.145845 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.145393    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2sn\" (UniqueName: \"kubernetes.io/projected/f34adf7f-7dff-4502-ba91-099f0bf0be62-kube-api-access-ht2sn\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:45.182796 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.180494    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:45.294215 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.294175    4827 scope.go:122] "RemoveContainer" containerID="d0a9cee90ec6e65a7d0b7754dcb6d8e025681b73c6d61c02812a486ac23f07e9"
Oct 29 22:59:45.373070 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.365592    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"]
Oct 29 22:59:45.473366 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.459091    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"]
Oct 29 22:59:45.473366 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.459216    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:45.543419 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.543311    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"]
Oct 29 22:59:45.563496 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.563465    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36588a39-cfmp5"]
Oct 29 22:59:45.574985 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.574959    4827 scope.go:122] "RemoveContainer" containerID="e05ddf44ef8b1faea72dd1533c512e1b92c94f0236c23af7e674633f71b875f9"
Oct 29 22:59:45.621786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.618421    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfm2\" (UniqueName: \"kubernetes.io/projected/6823014e-1288-465d-b69e-0d74fc8a6bc0-kube-api-access-6xfm2\") pod \"payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:45.621786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.618501    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823014e-1288-465d-b69e-0d74fc8a6bc0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:45.726856 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.726823    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfm2\" (UniqueName: \"kubernetes.io/projected/6823014e-1288-465d-b69e-0d74fc8a6bc0-kube-api-access-6xfm2\") pod \"payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:45.727021 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.726890    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823014e-1288-465d-b69e-0d74fc8a6bc0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:45.727894 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.727844    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823014e-1288-465d-b69e-0d74fc8a6bc0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:45.750867 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.750724    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfm2\" (UniqueName: \"kubernetes.io/projected/6823014e-1288-465d-b69e-0d74fc8a6bc0-kube-api-access-6xfm2\") pod \"payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:45.915103 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.915072    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"]
Oct 29 22:59:45.915349 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:45.915323    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34adf7f_7dff_4502_ba91_099f0bf0be62.slice/crio-9153b62e28d5bfce38fcaf07b8afda6801c72fafe519b00a2afd17cfdbf94c45 WatchSource:0}: Error finding container 9153b62e28d5bfce38fcaf07b8afda6801c72fafe519b00a2afd17cfdbf94c45: Status 404 returned error can't find the container with id 9153b62e28d5bfce38fcaf07b8afda6801c72fafe519b00a2afd17cfdbf94c45
Oct 29 22:59:45.962425 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:45.961850    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:46.501871 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.493243    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3319ba61-f527-4b57-af00-ede4288da8dd" path="/var/lib/kubelet/pods/3319ba61-f527-4b57-af00-ede4288da8dd/volumes"
Oct 29 22:59:46.588924 ip-10-0-134-205 kubenswrapper[4827]: W1029 22:59:46.588879    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6823014e_1288_465d_b69e_0d74fc8a6bc0.slice/crio-73020aad13d5ceacaf2ffd1f7ea63e4610da7eb5fc1859c258924776a7091de1 WatchSource:0}: Error finding container 73020aad13d5ceacaf2ffd1f7ea63e4610da7eb5fc1859c258924776a7091de1: Status 404 returned error can't find the container with id 73020aad13d5ceacaf2ffd1f7ea63e4610da7eb5fc1859c258924776a7091de1
Oct 29 22:59:46.590179 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.590157    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"]
Oct 29 22:59:46.733282 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.733245    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p" event={"ID":"f34adf7f-7dff-4502-ba91-099f0bf0be62","Type":"ContainerStarted","Data":"dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb"}
Oct 29 22:59:46.733465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.733314    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:46.733465 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.733339    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p" event={"ID":"f34adf7f-7dff-4502-ba91-099f0bf0be62","Type":"ContainerStarted","Data":"9153b62e28d5bfce38fcaf07b8afda6801c72fafe519b00a2afd17cfdbf94c45"}
Oct 29 22:59:46.735061 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.735036    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j" event={"ID":"6823014e-1288-465d-b69e-0d74fc8a6bc0","Type":"ContainerStarted","Data":"c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867"}
Oct 29 22:59:46.735191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.735078    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j" event={"ID":"6823014e-1288-465d-b69e-0d74fc8a6bc0","Type":"ContainerStarted","Data":"73020aad13d5ceacaf2ffd1f7ea63e4610da7eb5fc1859c258924776a7091de1"}
Oct 29 22:59:46.735191 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.735106    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:46.775926 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.775846    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p" podStartSLOduration=2.775832109 podStartE2EDuration="2.775832109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:46.772625208 +0000 UTC m=+1022.395536228" watchObservedRunningTime="2026-10-29 22:59:46.775832109 +0000 UTC m=+1022.398743129"
Oct 29 22:59:46.815980 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:46.815937    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j" podStartSLOduration=2.815924651 podStartE2EDuration="2.815924651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:46.812123929 +0000 UTC m=+1022.435034950" watchObservedRunningTime="2026-10-29 22:59:46.815924651 +0000 UTC m=+1022.438835701"
Oct 29 22:59:55.245786 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:55.244940    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"]
Oct 29 22:59:55.254396 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:55.254224    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-27d16a-gvm74"]
Oct 29 22:59:56.490250 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:56.490219    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5757d0ff-f91b-4b0f-bd12-bfca8bd6952c" path="/var/lib/kubelet/pods/5757d0ff-f91b-4b0f-bd12-bfca8bd6952c/volumes"
Oct 29 22:59:57.747515 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:57.747442    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 22:59:57.747515 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:57.747504    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 22:59:59.030159 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:59.030126    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"]
Oct 29 22:59:59.041554 ip-10-0-134-205 kubenswrapper[4827]: I1029 22:59:59.041528    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeadf9d4-vmxpb"]
Oct 29 23:00:00.165265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.165233    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"]
Oct 29 23:00:00.171736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.171714    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"]
Oct 29 23:00:00.171896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.171876    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.177324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.177298    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.194285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.194255    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"]
Oct 29 23:00:00.197481 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.197457    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:00:00.202127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.202081    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"]
Oct 29 23:00:00.209153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.209128    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.211557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.211530    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"]
Oct 29 23:00:00.222314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.219599    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.222314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.220615    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"]
Oct 29 23:00:00.233083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233054    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd33576d-8fab-40c9-8690-0f770df77547-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.233227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233129    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2kf\" (UniqueName: \"kubernetes.io/projected/cd33576d-8fab-40c9-8690-0f770df77547-kube-api-access-7h2kf\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.233227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233167    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cww5j\" (UniqueName: \"kubernetes.io/projected/117c6362-84ed-4158-a670-9404fa625943-kube-api-access-cww5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.233227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233206    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsk5\" (UniqueName: \"kubernetes.io/projected/273ae80a-cc37-4511-9e1a-485554e20f43-kube-api-access-wgsk5\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.234271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233393    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rh79\" (UniqueName: \"kubernetes.io/projected/cbe05cd7-9c9d-455a-8a8b-69c903af653e-kube-api-access-8rh79\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.234271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233469    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe05cd7-9c9d-455a-8a8b-69c903af653e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.234271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233552    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ae80a-cc37-4511-9e1a-485554e20f43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.234271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233675    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117c6362-84ed-4158-a670-9404fa625943-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.234271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.233871    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"]
Oct 29 23:00:00.240839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.240143    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.245173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.245150    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"]
Oct 29 23:00:00.258470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.258441    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"]
Oct 29 23:00:00.258649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.258575    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.263573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.263540    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"]
Oct 29 23:00:00.270698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.270669    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.273360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.273339    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"]
Oct 29 23:00:00.279998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.279964    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"]
Oct 29 23:00:00.293724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.293695    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"]
Oct 29 23:00:00.302323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.302292    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"]
Oct 29 23:00:00.335409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.335367    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe05cd7-9c9d-455a-8a8b-69c903af653e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.335594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.335463    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqrr\" (UniqueName: \"kubernetes.io/projected/a6419ba1-4c1c-4a70-b654-183fa00dbd27-kube-api-access-bfqrr\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.335594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.335515    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e8482-aaa4-43c6-bf67-01087c5675b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.335679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.335588    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ae80a-cc37-4511-9e1a-485554e20f43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.335679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.335661    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6419ba1-4c1c-4a70-b654-183fa00dbd27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.335888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.335863    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117c6362-84ed-4158-a670-9404fa625943-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.335943 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.335907    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece158c-1f91-4810-9c88-df9aaf8ef8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.336498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.336466    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6lq\" (UniqueName: \"kubernetes.io/projected/9c9e8482-aaa4-43c6-bf67-01087c5675b1-kube-api-access-zw6lq\") pod \"maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.336585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.336571    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd33576d-8fab-40c9-8690-0f770df77547-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337027    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2kf\" (UniqueName: \"kubernetes.io/projected/cd33576d-8fab-40c9-8690-0f770df77547-kube-api-access-7h2kf\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337065    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ae80a-cc37-4511-9e1a-485554e20f43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337092    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cww5j\" (UniqueName: \"kubernetes.io/projected/117c6362-84ed-4158-a670-9404fa625943-kube-api-access-cww5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337149    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsk5\" (UniqueName: \"kubernetes.io/projected/273ae80a-cc37-4511-9e1a-485554e20f43-kube-api-access-wgsk5\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337259    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj485\" (UniqueName: \"kubernetes.io/projected/fece158c-1f91-4810-9c88-df9aaf8ef8be-kube-api-access-cj485\") pod \"maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337277    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117c6362-84ed-4158-a670-9404fa625943-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337411    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd33576d-8fab-40c9-8690-0f770df77547-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.337622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.337463    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rh79\" (UniqueName: \"kubernetes.io/projected/cbe05cd7-9c9d-455a-8a8b-69c903af653e-kube-api-access-8rh79\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.348859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.348826    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe05cd7-9c9d-455a-8a8b-69c903af653e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.360677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.360651    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsk5\" (UniqueName: \"kubernetes.io/projected/273ae80a-cc37-4511-9e1a-485554e20f43-kube-api-access-wgsk5\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.361003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.360972    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2kf\" (UniqueName: \"kubernetes.io/projected/cd33576d-8fab-40c9-8690-0f770df77547-kube-api-access-7h2kf\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.361156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.361137    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rh79\" (UniqueName: \"kubernetes.io/projected/cbe05cd7-9c9d-455a-8a8b-69c903af653e-kube-api-access-8rh79\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.361231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.361209    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cww5j\" (UniqueName: \"kubernetes.io/projected/117c6362-84ed-4158-a670-9404fa625943-kube-api-access-cww5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.439317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.439239    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqrr\" (UniqueName: \"kubernetes.io/projected/a6419ba1-4c1c-4a70-b654-183fa00dbd27-kube-api-access-bfqrr\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.439317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.439287    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e8482-aaa4-43c6-bf67-01087c5675b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.439317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.439307    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6419ba1-4c1c-4a70-b654-183fa00dbd27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.439604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.439465    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece158c-1f91-4810-9c88-df9aaf8ef8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.439604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.439505    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6lq\" (UniqueName: \"kubernetes.io/projected/9c9e8482-aaa4-43c6-bf67-01087c5675b1-kube-api-access-zw6lq\") pod \"maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.439721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.439689    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj485\" (UniqueName: \"kubernetes.io/projected/fece158c-1f91-4810-9c88-df9aaf8ef8be-kube-api-access-cj485\") pod \"maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.440072 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.440036    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e8482-aaa4-43c6-bf67-01087c5675b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.440192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.440089    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6419ba1-4c1c-4a70-b654-183fa00dbd27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.440242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.440218    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece158c-1f91-4810-9c88-df9aaf8ef8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.463002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.462976    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6lq\" (UniqueName: \"kubernetes.io/projected/9c9e8482-aaa4-43c6-bf67-01087c5675b1-kube-api-access-zw6lq\") pod \"maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.463178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.463064    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj485\" (UniqueName: \"kubernetes.io/projected/fece158c-1f91-4810-9c88-df9aaf8ef8be-kube-api-access-cj485\") pod \"maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.463178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.463117    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqrr\" (UniqueName: \"kubernetes.io/projected/a6419ba1-4c1c-4a70-b654-183fa00dbd27-kube-api-access-bfqrr\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.491315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.491286    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b62ece1-6022-4ffe-88c5-8062881fa212" path="/var/lib/kubelet/pods/0b62ece1-6022-4ffe-88c5-8062881fa212/volumes"
Oct 29 23:00:00.491928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.491907    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:00.502641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.502619    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:00.544920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.544888    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:00.562272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.562016    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:00.582238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.581707    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:00.591825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.591793    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:00.598807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.598695    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:00.816325 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:00.816291    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ae80a_cc37_4511_9e1a_485554e20f43.slice/crio-e02b3a1763bbb3906ca74ae2adc65813fb9f1884e4332815e681417b0f9830b8 WatchSource:0}: Error finding container e02b3a1763bbb3906ca74ae2adc65813fb9f1884e4332815e681417b0f9830b8: Status 404 returned error can't find the container with id e02b3a1763bbb3906ca74ae2adc65813fb9f1884e4332815e681417b0f9830b8
Oct 29 23:00:00.817716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.817693    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"]
Oct 29 23:00:00.829315 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:00.829290    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117c6362_84ed_4158_a670_9404fa625943.slice/crio-97507001b44cf3f66727f869d41914b625a2a5211c39b51d9a2388eb27f7bca1 WatchSource:0}: Error finding container 97507001b44cf3f66727f869d41914b625a2a5211c39b51d9a2388eb27f7bca1: Status 404 returned error can't find the container with id 97507001b44cf3f66727f869d41914b625a2a5211c39b51d9a2388eb27f7bca1
Oct 29 23:00:00.833048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.833021    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"]
Oct 29 23:00:00.988324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:00.988093    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"]
Oct 29 23:00:01.040099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.040063    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"]
Oct 29 23:00:01.082457 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:01.082424    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfece158c_1f91_4810_9c88_df9aaf8ef8be.slice/crio-500aa67eddc369e83545a553c4d17b5d297e9592a1b9c0920494db626c1446eb WatchSource:0}: Error finding container 500aa67eddc369e83545a553c4d17b5d297e9592a1b9c0920494db626c1446eb: Status 404 returned error can't find the container with id 500aa67eddc369e83545a553c4d17b5d297e9592a1b9c0920494db626c1446eb
Oct 29 23:00:01.179741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.178893    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"]
Oct 29 23:00:01.222790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.222751    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"]
Oct 29 23:00:01.231710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.231677    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"]
Oct 29 23:00:01.270368 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:01.270170    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd33576d_8fab_40c9_8690_0f770df77547.slice/crio-f7fc0af8b2983b30fc8a367c25e0df356039336f1b9df97fcf0330339913b166 WatchSource:0}: Error finding container f7fc0af8b2983b30fc8a367c25e0df356039336f1b9df97fcf0330339913b166: Status 404 returned error can't find the container with id f7fc0af8b2983b30fc8a367c25e0df356039336f1b9df97fcf0330339913b166
Oct 29 23:00:01.270499 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:01.270472    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9e8482_aaa4_43c6_bf67_01087c5675b1.slice/crio-b9fdfe47910e3596a6d7b9793715cb85740ef15d8f0ce7761a9519e1e0626c22 WatchSource:0}: Error finding container b9fdfe47910e3596a6d7b9793715cb85740ef15d8f0ce7761a9519e1e0626c22: Status 404 returned error can't find the container with id b9fdfe47910e3596a6d7b9793715cb85740ef15d8f0ce7761a9519e1e0626c22
Oct 29 23:00:01.271526 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:01.271501    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6419ba1_4c1c_4a70_b654_183fa00dbd27.slice/crio-473065d8f02302e4427122188ad330b626bd5a40d9a7aacf3a2d8207b9610866 WatchSource:0}: Error finding container 473065d8f02302e4427122188ad330b626bd5a40d9a7aacf3a2d8207b9610866: Status 404 returned error can't find the container with id 473065d8f02302e4427122188ad330b626bd5a40d9a7aacf3a2d8207b9610866
Oct 29 23:00:01.392760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.392728    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" event={"ID":"273ae80a-cc37-4511-9e1a-485554e20f43","Type":"ContainerStarted","Data":"e02b3a1763bbb3906ca74ae2adc65813fb9f1884e4332815e681417b0f9830b8"}
Oct 29 23:00:01.395072 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.395040    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk" event={"ID":"9c9e8482-aaa4-43c6-bf67-01087c5675b1","Type":"ContainerStarted","Data":"b9fdfe47910e3596a6d7b9793715cb85740ef15d8f0ce7761a9519e1e0626c22"}
Oct 29 23:00:01.396296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.396255    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" event={"ID":"a6419ba1-4c1c-4a70-b654-183fa00dbd27","Type":"ContainerStarted","Data":"473065d8f02302e4427122188ad330b626bd5a40d9a7aacf3a2d8207b9610866"}
Oct 29 23:00:01.397822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.397795    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" event={"ID":"fece158c-1f91-4810-9c88-df9aaf8ef8be","Type":"ContainerStarted","Data":"500aa67eddc369e83545a553c4d17b5d297e9592a1b9c0920494db626c1446eb"}
Oct 29 23:00:01.399256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.399233    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" event={"ID":"cbe05cd7-9c9d-455a-8a8b-69c903af653e","Type":"ContainerStarted","Data":"08ff8da91ec5119eedfd1809d00ee764cc75954c316df3dfed54be2d08f8ff75"}
Oct 29 23:00:01.400898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.400869    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" event={"ID":"cd33576d-8fab-40c9-8690-0f770df77547","Type":"ContainerStarted","Data":"f7fc0af8b2983b30fc8a367c25e0df356039336f1b9df97fcf0330339913b166"}
Oct 29 23:00:01.402321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:01.402301    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" event={"ID":"117c6362-84ed-4158-a670-9404fa625943","Type":"ContainerStarted","Data":"97507001b44cf3f66727f869d41914b625a2a5211c39b51d9a2388eb27f7bca1"}
Oct 29 23:00:02.413154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.413122    4827 generic.go:356] "Generic (PLEG): container finished" podID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerID="0b3ec93c2f1320bf6746d0d73a748c42ffa1fb4d38df82ddee44d37090ea36ef" exitCode=60
Oct 29 23:00:02.413558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.413210    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" event={"ID":"fece158c-1f91-4810-9c88-df9aaf8ef8be","Type":"ContainerDied","Data":"0b3ec93c2f1320bf6746d0d73a748c42ffa1fb4d38df82ddee44d37090ea36ef"}
Oct 29 23:00:02.413603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.413570    4827 scope.go:122] "RemoveContainer" containerID="0b3ec93c2f1320bf6746d0d73a748c42ffa1fb4d38df82ddee44d37090ea36ef"
Oct 29 23:00:02.415247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.415221    4827 generic.go:356] "Generic (PLEG): container finished" podID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerID="473ae8786db5ae8f2fe47a926869a6f5d73c61f6d3514d992c526f6dadb85cc9" exitCode=60
Oct 29 23:00:02.415349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.415292    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" event={"ID":"cbe05cd7-9c9d-455a-8a8b-69c903af653e","Type":"ContainerDied","Data":"473ae8786db5ae8f2fe47a926869a6f5d73c61f6d3514d992c526f6dadb85cc9"}
Oct 29 23:00:02.415589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.415568    4827 scope.go:122] "RemoveContainer" containerID="473ae8786db5ae8f2fe47a926869a6f5d73c61f6d3514d992c526f6dadb85cc9"
Oct 29 23:00:02.417282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.417262    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" event={"ID":"cd33576d-8fab-40c9-8690-0f770df77547","Type":"ContainerStarted","Data":"f84dd4e1fe4b6df94c28772bf2c2174c547f0252c547e7ed5fe69490984546c4"}
Oct 29 23:00:02.420894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.420866    4827 generic.go:356] "Generic (PLEG): container finished" podID="117c6362-84ed-4158-a670-9404fa625943" containerID="5efa99940649f15b143732491ed2fc1e6a32b9d045916b603f7468effbb36a21" exitCode=60
Oct 29 23:00:02.421401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.420990    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" event={"ID":"117c6362-84ed-4158-a670-9404fa625943","Type":"ContainerDied","Data":"5efa99940649f15b143732491ed2fc1e6a32b9d045916b603f7468effbb36a21"}
Oct 29 23:00:02.421401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.421264    4827 scope.go:122] "RemoveContainer" containerID="5efa99940649f15b143732491ed2fc1e6a32b9d045916b603f7468effbb36a21"
Oct 29 23:00:02.422847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.422827    4827 generic.go:356] "Generic (PLEG): container finished" podID="273ae80a-cc37-4511-9e1a-485554e20f43" containerID="96ba1555fa538bd3f2b9e5cafc07fd7e8af5ed8dbad4702cbbfd2c5315eb9dea" exitCode=60
Oct 29 23:00:02.422943 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.422850    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" event={"ID":"273ae80a-cc37-4511-9e1a-485554e20f43","Type":"ContainerDied","Data":"96ba1555fa538bd3f2b9e5cafc07fd7e8af5ed8dbad4702cbbfd2c5315eb9dea"}
Oct 29 23:00:02.423129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.423115    4827 scope.go:122] "RemoveContainer" containerID="96ba1555fa538bd3f2b9e5cafc07fd7e8af5ed8dbad4702cbbfd2c5315eb9dea"
Oct 29 23:00:02.424989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.424966    4827 generic.go:356] "Generic (PLEG): container finished" podID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerID="2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80" exitCode=60
Oct 29 23:00:02.425086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.425045    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk" event={"ID":"9c9e8482-aaa4-43c6-bf67-01087c5675b1","Type":"ContainerDied","Data":"2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80"}
Oct 29 23:00:02.425256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.425240    4827 scope.go:122] "RemoveContainer" containerID="2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80"
Oct 29 23:00:02.427284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.427260    4827 generic.go:356] "Generic (PLEG): container finished" podID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerID="1f399d2f4864057747eca9137b38dce40915260be8cc081cdc754c07cf9cb422" exitCode=60
Oct 29 23:00:02.427396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.427329    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" event={"ID":"a6419ba1-4c1c-4a70-b654-183fa00dbd27","Type":"ContainerDied","Data":"1f399d2f4864057747eca9137b38dce40915260be8cc081cdc754c07cf9cb422"}
Oct 29 23:00:02.428089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.427547    4827 scope.go:122] "RemoveContainer" containerID="1f399d2f4864057747eca9137b38dce40915260be8cc081cdc754c07cf9cb422"
Oct 29 23:00:02.803975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:02.803924    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" podStartSLOduration=2.803911779 podStartE2EDuration="2.803911779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.799605186 +0000 UTC m=+1038.422516225" watchObservedRunningTime="2026-10-29 23:00:02.803911779 +0000 UTC m=+1038.426822825"
Oct 29 23:00:02.891593 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:02.891555    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:00:03.442876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.442827    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" event={"ID":"cbe05cd7-9c9d-455a-8a8b-69c903af653e","Type":"ContainerStarted","Data":"b1847ad5c731435c3905eb36d32adb195be3c9eed02e14a984c5e1bac2b1f155"}
Oct 29 23:00:03.445068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.444858    4827 generic.go:356] "Generic (PLEG): container finished" podID="cd33576d-8fab-40c9-8690-0f770df77547" containerID="f84dd4e1fe4b6df94c28772bf2c2174c547f0252c547e7ed5fe69490984546c4" exitCode=60
Oct 29 23:00:03.445068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.444886    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" event={"ID":"cd33576d-8fab-40c9-8690-0f770df77547","Type":"ContainerDied","Data":"f84dd4e1fe4b6df94c28772bf2c2174c547f0252c547e7ed5fe69490984546c4"}
Oct 29 23:00:03.448251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.448228    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" event={"ID":"117c6362-84ed-4158-a670-9404fa625943","Type":"ContainerStarted","Data":"c55d40aafc029e0c4834e09ab4aa1ba4adcea4215d2e03d759a47c2d86dd8e68"}
Oct 29 23:00:03.449964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.449533    4827 scope.go:122] "RemoveContainer" containerID="f84dd4e1fe4b6df94c28772bf2c2174c547f0252c547e7ed5fe69490984546c4"
Oct 29 23:00:03.451470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.451444    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" event={"ID":"273ae80a-cc37-4511-9e1a-485554e20f43","Type":"ContainerStarted","Data":"399ff610ed52a2812e60197bfce6bef5bbe112f8e242df5bd7757d3a342eb920"}
Oct 29 23:00:03.478467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.466660    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk" event={"ID":"9c9e8482-aaa4-43c6-bf67-01087c5675b1","Type":"ContainerStarted","Data":"401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289"}
Oct 29 23:00:03.478467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.471075    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" event={"ID":"a6419ba1-4c1c-4a70-b654-183fa00dbd27","Type":"ContainerStarted","Data":"80f8fbe0666278fbfb642fbc7e753ef03498e887b18601c933d24d373280a9b6"}
Oct 29 23:00:03.478467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.474689    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" event={"ID":"fece158c-1f91-4810-9c88-df9aaf8ef8be","Type":"ContainerStarted","Data":"c65876ef219ae1300337932e397e54582702b23166295bccb8cbe2f4a405abf0"}
Oct 29 23:00:03.482282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.482253    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"]
Oct 29 23:00:03.482560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.482533    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy" containerID="cri-o://29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900" gracePeriod=30
Oct 29 23:00:03.498833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.498757    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" podStartSLOduration=3.49873781 podStartE2EDuration="3.49873781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.495310914 +0000 UTC m=+1039.118221933" watchObservedRunningTime="2026-10-29 23:00:03.49873781 +0000 UTC m=+1039.121648831"
Oct 29 23:00:03.543016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.542958    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" podStartSLOduration=3.542940396 podStartE2EDuration="3.542940396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.536136487 +0000 UTC m=+1039.159047508" watchObservedRunningTime="2026-10-29 23:00:03.542940396 +0000 UTC m=+1039.165851416"
Oct 29 23:00:03.600925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.600862    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" podStartSLOduration=3.600840691 podStartE2EDuration="3.600840691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.591239357 +0000 UTC m=+1039.214150404" watchObservedRunningTime="2026-10-29 23:00:03.600840691 +0000 UTC m=+1039.223751728"
Oct 29 23:00:03.757798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:03.744634    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" podStartSLOduration=3.744612191 podStartE2EDuration="3.744612191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.708824102 +0000 UTC m=+1039.331735123" watchObservedRunningTime="2026-10-29 23:00:03.744612191 +0000 UTC m=+1039.367523212"
Oct 29 23:00:04.167825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.160709    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk" podStartSLOduration=4.160686784 podStartE2EDuration="4.160686784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.831487897 +0000 UTC m=+1039.454398918" watchObservedRunningTime="2026-10-29 23:00:04.160686784 +0000 UTC m=+1039.783597839"
Oct 29 23:00:04.673789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.673658    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"]
Oct 29 23:00:04.680617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.680553    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" podStartSLOduration=4.680534795 podStartE2EDuration="4.680534795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.274173456 +0000 UTC m=+1039.897084496" watchObservedRunningTime="2026-10-29 23:00:04.680534795 +0000 UTC m=+1040.303445815"
Oct 29 23:00:04.856833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.846472    4827 generic.go:356] "Generic (PLEG): container finished" podID="273ae80a-cc37-4511-9e1a-485554e20f43" containerID="399ff610ed52a2812e60197bfce6bef5bbe112f8e242df5bd7757d3a342eb920" exitCode=60
Oct 29 23:00:04.856833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.846693    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" event={"ID":"273ae80a-cc37-4511-9e1a-485554e20f43","Type":"ContainerDied","Data":"399ff610ed52a2812e60197bfce6bef5bbe112f8e242df5bd7757d3a342eb920"}
Oct 29 23:00:04.856833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.846752    4827 scope.go:122] "RemoveContainer" containerID="96ba1555fa538bd3f2b9e5cafc07fd7e8af5ed8dbad4702cbbfd2c5315eb9dea"
Oct 29 23:00:04.856833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.847453    4827 scope.go:122] "RemoveContainer" containerID="399ff610ed52a2812e60197bfce6bef5bbe112f8e242df5bd7757d3a342eb920"
Oct 29 23:00:04.856833 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:04.848017    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs_odh-ai-gateway-infra(273ae80a-cc37-4511-9e1a-485554e20f43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" podUID="273ae80a-cc37-4511-9e1a-485554e20f43"
Oct 29 23:00:04.915832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.902108    4827 generic.go:356] "Generic (PLEG): container finished" podID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerID="c65876ef219ae1300337932e397e54582702b23166295bccb8cbe2f4a405abf0" exitCode=60
Oct 29 23:00:04.915832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.902336    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" event={"ID":"fece158c-1f91-4810-9c88-df9aaf8ef8be","Type":"ContainerDied","Data":"c65876ef219ae1300337932e397e54582702b23166295bccb8cbe2f4a405abf0"}
Oct 29 23:00:04.915832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.902894    4827 scope.go:122] "RemoveContainer" containerID="c65876ef219ae1300337932e397e54582702b23166295bccb8cbe2f4a405abf0"
Oct 29 23:00:04.915832 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:04.903496    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j_odh-ai-gateway-infra(fece158c-1f91-4810-9c88-df9aaf8ef8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be"
Oct 29 23:00:04.948882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.947139    4827 generic.go:356] "Generic (PLEG): container finished" podID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerID="b1847ad5c731435c3905eb36d32adb195be3c9eed02e14a984c5e1bac2b1f155" exitCode=60
Oct 29 23:00:04.948882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.947328    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" event={"ID":"cbe05cd7-9c9d-455a-8a8b-69c903af653e","Type":"ContainerDied","Data":"b1847ad5c731435c3905eb36d32adb195be3c9eed02e14a984c5e1bac2b1f155"}
Oct 29 23:00:04.948882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:04.948021    4827 scope.go:122] "RemoveContainer" containerID="b1847ad5c731435c3905eb36d32adb195be3c9eed02e14a984c5e1bac2b1f155"
Oct 29 23:00:04.954800 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:04.948560    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t_odh-ai-gateway-infra(cbe05cd7-9c9d-455a-8a8b-69c903af653e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e"
Oct 29 23:00:05.222794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:05.222668    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"]
Oct 29 23:00:05.239506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:05.238481    4827 scope.go:122] "RemoveContainer" containerID="0b3ec93c2f1320bf6746d0d73a748c42ffa1fb4d38df82ddee44d37090ea36ef"
Oct 29 23:00:05.357019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:05.343135    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd525f-mmp4z"]
Oct 29 23:00:05.863600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:05.818992    4827 scope.go:122] "RemoveContainer" containerID="473ae8786db5ae8f2fe47a926869a6f5d73c61f6d3514d992c526f6dadb85cc9"
Oct 29 23:00:06.080869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.070017    4827 scope.go:122] "RemoveContainer" containerID="399ff610ed52a2812e60197bfce6bef5bbe112f8e242df5bd7757d3a342eb920"
Oct 29 23:00:06.080869 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:06.070783    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs_odh-ai-gateway-infra(273ae80a-cc37-4511-9e1a-485554e20f43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" podUID="273ae80a-cc37-4511-9e1a-485554e20f43"
Oct 29 23:00:06.080869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.078882    4827 generic.go:356] "Generic (PLEG): container finished" podID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerID="80f8fbe0666278fbfb642fbc7e753ef03498e887b18601c933d24d373280a9b6" exitCode=60
Oct 29 23:00:06.080869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.078992    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" event={"ID":"a6419ba1-4c1c-4a70-b654-183fa00dbd27","Type":"ContainerDied","Data":"80f8fbe0666278fbfb642fbc7e753ef03498e887b18601c933d24d373280a9b6"}
Oct 29 23:00:06.080869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.079045    4827 scope.go:122] "RemoveContainer" containerID="1f399d2f4864057747eca9137b38dce40915260be8cc081cdc754c07cf9cb422"
Oct 29 23:00:06.080869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.079631    4827 scope.go:122] "RemoveContainer" containerID="80f8fbe0666278fbfb642fbc7e753ef03498e887b18601c933d24d373280a9b6"
Oct 29 23:00:06.080869 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:06.080140    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8_odh-ai-gateway-infra(a6419ba1-4c1c-4a70-b654-183fa00dbd27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27"
Oct 29 23:00:06.096804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.089251    4827 scope.go:122] "RemoveContainer" containerID="c65876ef219ae1300337932e397e54582702b23166295bccb8cbe2f4a405abf0"
Oct 29 23:00:06.096804 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:06.089743    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j_odh-ai-gateway-infra(fece158c-1f91-4810-9c88-df9aaf8ef8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be"
Oct 29 23:00:06.098699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.098539    4827 scope.go:122] "RemoveContainer" containerID="b1847ad5c731435c3905eb36d32adb195be3c9eed02e14a984c5e1bac2b1f155"
Oct 29 23:00:06.110458 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:06.099294    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t_odh-ai-gateway-infra(cbe05cd7-9c9d-455a-8a8b-69c903af653e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e"
Oct 29 23:00:06.110458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.107778    4827 generic.go:356] "Generic (PLEG): container finished" podID="cd33576d-8fab-40c9-8690-0f770df77547" containerID="91980bf633b8ae8e68aac8d5a2c6d7d226fe37f82c3bc07e5e2a1940a3fec74b" exitCode=60
Oct 29 23:00:06.110458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.107858    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" event={"ID":"cd33576d-8fab-40c9-8690-0f770df77547","Type":"ContainerDied","Data":"91980bf633b8ae8e68aac8d5a2c6d7d226fe37f82c3bc07e5e2a1940a3fec74b"}
Oct 29 23:00:06.110458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.108355    4827 scope.go:122] "RemoveContainer" containerID="91980bf633b8ae8e68aac8d5a2c6d7d226fe37f82c3bc07e5e2a1940a3fec74b"
Oct 29 23:00:06.110458 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:06.108812    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m_odh-ai-gateway-infra(cd33576d-8fab-40c9-8690-0f770df77547)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" podUID="cd33576d-8fab-40c9-8690-0f770df77547"
Oct 29 23:00:06.132059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.132014    4827 generic.go:356] "Generic (PLEG): container finished" podID="117c6362-84ed-4158-a670-9404fa625943" containerID="c55d40aafc029e0c4834e09ab4aa1ba4adcea4215d2e03d759a47c2d86dd8e68" exitCode=60
Oct 29 23:00:06.139450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.139419    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerName="cleanup" containerID="cri-o://401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289" gracePeriod=30
Oct 29 23:00:06.139822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.132268    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" event={"ID":"117c6362-84ed-4158-a670-9404fa625943","Type":"ContainerDied","Data":"c55d40aafc029e0c4834e09ab4aa1ba4adcea4215d2e03d759a47c2d86dd8e68"}
Oct 29 23:00:06.140544 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.140477    4827 scope.go:122] "RemoveContainer" containerID="c55d40aafc029e0c4834e09ab4aa1ba4adcea4215d2e03d759a47c2d86dd8e68"
Oct 29 23:00:06.150488 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:06.141927    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl_odh-ai-gateway-infra(117c6362-84ed-4158-a670-9404fa625943)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" podUID="117c6362-84ed-4158-a670-9404fa625943"
Oct 29 23:00:06.172350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.172237    4827 scope.go:122] "RemoveContainer" containerID="f84dd4e1fe4b6df94c28772bf2c2174c547f0252c547e7ed5fe69490984546c4"
Oct 29 23:00:06.204630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.204606    4827 scope.go:122] "RemoveContainer" containerID="5efa99940649f15b143732491ed2fc1e6a32b9d045916b603f7468effbb36a21"
Oct 29 23:00:06.491107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:06.491079    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6e2b2a-3a3a-4660-bae2-aaf5384b4243" path="/var/lib/kubelet/pods/ec6e2b2a-3a3a-4660-bae2-aaf5384b4243/volumes"
Oct 29 23:00:08.806129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.806100    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 23:00:08.921164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921127    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8923c498-8a01-4288-85ae-8f71b92545bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8923c498-8a01-4288-85ae-8f71b92545bf-istio-podinfo\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921202    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8923c498-8a01-4288-85ae-8f71b92545bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8923c498-8a01-4288-85ae-8f71b92545bf-istiod-ca-cert\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921246    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-data\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921319    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-kube-api-access-gdjf4\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-kube-api-access-gdjf4\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921408    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-certs\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921592    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-credential-socket\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921729    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-socket\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921857    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-istio-token\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-istio-token\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.921914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.921896    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-envoy\") pod \"8923c498-8a01-4288-85ae-8f71b92545bf\" (UID: \"8923c498-8a01-4288-85ae-8f71b92545bf\") "
Oct 29 23:00:08.922358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.922241    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8923c498-8a01-4288-85ae-8f71b92545bf-istiod-ca-cert" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:08.922720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.922589    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-credential-socket" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:08.923045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.923024    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8923c498-8a01-4288-85ae-8f71b92545bf-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:08.923126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.923051    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:08.923126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.923115    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-socket" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:08.923232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.923177    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-certs" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:08.925307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.925270    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8923c498-8a01-4288-85ae-8f71b92545bf-istio-podinfo" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:08.925409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.925315    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-data" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:08.925634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.925592    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-kube-api-access-gdjf4" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "kube-api-access-gdjf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:08.926237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.926216    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-envoy" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:08.928262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:08.928224    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-istio-token" pod "8923c498-8a01-4288-85ae-8f71b92545bf" (UID: "8923c498-8a01-4288-85ae-8f71b92545bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:09.025538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.025455    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8923c498-8a01-4288-85ae-8f71b92545bf-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:09.025538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.025484    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:09.025538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.025494    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdjf4\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-kube-api-access-gdjf4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:09.025538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.025505    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:09.025538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.025514    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:09.025538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.025522    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8923c498-8a01-4288-85ae-8f71b92545bf-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:09.025538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.025529    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8923c498-8a01-4288-85ae-8f71b92545bf-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:09.045748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.045709    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"]
Oct 29 23:00:09.051099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.051073    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f0b7366-mzkfc"]
Oct 29 23:00:09.174926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.174894    4827 generic.go:356] "Generic (PLEG): container finished" podID="8923c498-8a01-4288-85ae-8f71b92545bf" containerID="29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900" exitCode=0
Oct 29 23:00:09.175116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.174953    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"
Oct 29 23:00:09.175116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.174973    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" event={"ID":"8923c498-8a01-4288-85ae-8f71b92545bf","Type":"ContainerDied","Data":"29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900"}
Oct 29 23:00:09.175116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.175029    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld" event={"ID":"8923c498-8a01-4288-85ae-8f71b92545bf","Type":"ContainerDied","Data":"340f3c2b38e448f7a4a41edb513421eb1977c473f3047694a381b66708322241"}
Oct 29 23:00:09.175116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.175052    4827 scope.go:122] "RemoveContainer" containerID="29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900"
Oct 29 23:00:09.189028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.189006    4827 scope.go:122] "RemoveContainer" containerID="6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a"
Oct 29 23:00:09.205854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.205832    4827 scope.go:122] "RemoveContainer" containerID="29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900"
Oct 29 23:00:09.206210 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:09.206163    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900\": container with ID starting with 29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900 not found: ID does not exist" containerID="29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900"
Oct 29 23:00:09.206210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.206202    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900"} err="failed to get container status \"29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900\": rpc error: code = NotFound desc = could not find container \"29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900\": container with ID starting with 29986380e1edbac9f44115008e598218f2b8e910918e570ed98c05bf37653900 not found: ID does not exist"
Oct 29 23:00:09.206407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.206231    4827 scope.go:122] "RemoveContainer" containerID="6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a"
Oct 29 23:00:09.206590 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:09.206560    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a\": container with ID starting with 6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a not found: ID does not exist" containerID="6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a"
Oct 29 23:00:09.206712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.206599    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a"} err="failed to get container status \"6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a\": rpc error: code = NotFound desc = could not find container \"6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a\": container with ID starting with 6d46da3fc9776ef6a885865bdd48b4cab0b9a1eb16a45a77577b436c80984b9a not found: ID does not exist"
Oct 29 23:00:09.250253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.250218    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"]
Oct 29 23:00:09.262823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:09.262733    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-5p8ld"]
Oct 29 23:00:10.491048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:10.491018    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" path="/var/lib/kubelet/pods/8923c498-8a01-4288-85ae-8f71b92545bf/volumes"
Oct 29 23:00:10.491675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:10.491658    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95342afb-e460-4454-86ff-460a10966b75" path="/var/lib/kubelet/pods/95342afb-e460-4454-86ff-460a10966b75/volumes"
Oct 29 23:00:11.094494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.008809    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"]
Oct 29 23:00:11.094494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.010106    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy"
Oct 29 23:00:11.094494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.010130    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy"
Oct 29 23:00:11.094494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.010146    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy"
Oct 29 23:00:11.094494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.010153    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy"
Oct 29 23:00:11.094494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.010373    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy"
Oct 29 23:00:11.094494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.010388    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8923c498-8a01-4288-85ae-8f71b92545bf" containerName="istio-proxy"
Oct 29 23:00:11.411019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.410984    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"]
Oct 29 23:00:11.411268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.411255    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"]
Oct 29 23:00:11.422825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.417416    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.437268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.437127    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f173b1\""
Oct 29 23:00:11.458255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.458225    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f173b1-dockercfg-rqzpc\""
Oct 29 23:00:11.495594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.495567    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:11.516840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.508091    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrqpg\" (UniqueName: \"kubernetes.io/projected/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-kube-api-access-jrqpg\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.516840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.508306    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.563832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.563167    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"]
Oct 29 23:00:11.621158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.621113    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrqpg\" (UniqueName: \"kubernetes.io/projected/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-kube-api-access-jrqpg\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.638174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.638139    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.638500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.638480    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr7jl\" (UniqueName: \"kubernetes.io/projected/f0e3f789-5057-4149-adae-d626175767e0-kube-api-access-hr7jl\") pod \"payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:11.638672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.638655    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e3f789-5057-4149-adae-d626175767e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:11.639864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.639811    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.685834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.681747    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrqpg\" (UniqueName: \"kubernetes.io/projected/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-kube-api-access-jrqpg\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.765838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.763493    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr7jl\" (UniqueName: \"kubernetes.io/projected/f0e3f789-5057-4149-adae-d626175767e0-kube-api-access-hr7jl\") pod \"payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:11.765838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.763584    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e3f789-5057-4149-adae-d626175767e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:11.765838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.764549    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:11.769791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.766815    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e3f789-5057-4149-adae-d626175767e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:11.813904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.810215    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr7jl\" (UniqueName: \"kubernetes.io/projected/f0e3f789-5057-4149-adae-d626175767e0-kube-api-access-hr7jl\") pod \"payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:11.878973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:11.875092    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:12.099798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.099712    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"]
Oct 29 23:00:12.100402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.100156    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy" containerID="cri-o://a0420fef29f16ea9a567ed0e191edd775d2741f83f8b94291269cd462d7515a5" gracePeriod=30
Oct 29 23:00:12.782351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.782234    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"]
Oct 29 23:00:12.791022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.789919    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.804075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.803904    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"]
Oct 29 23:00:12.804830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.804810    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:12.838463 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:12.838429    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0fa94f_5748_46f8_adeb_8ad243f3e72d.slice/crio-a8558edc71c8682974eb87691b5d36cbdade311ecb37a219550ee1dfab685ef9 WatchSource:0}: Error finding container a8558edc71c8682974eb87691b5d36cbdade311ecb37a219550ee1dfab685ef9: Status 404 returned error can't find the container with id a8558edc71c8682974eb87691b5d36cbdade311ecb37a219550ee1dfab685ef9
Oct 29 23:00:12.843037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.843016    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"]
Oct 29 23:00:12.867922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.867892    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.868126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.868107    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.868254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.868233    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9rx\" (UniqueName: \"kubernetes.io/projected/f146ab2e-c76c-46c7-8e0f-0379c50155da-kube-api-access-6x9rx\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.969689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.969655    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.969876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.969857    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.969994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.969980    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9rx\" (UniqueName: \"kubernetes.io/projected/f146ab2e-c76c-46c7-8e0f-0379c50155da-kube-api-access-6x9rx\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.970700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.970675    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.972054 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.972033    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:12.993994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:12.993973    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9rx\" (UniqueName: \"kubernetes.io/projected/f146ab2e-c76c-46c7-8e0f-0379c50155da-kube-api-access-6x9rx\") pod \"maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:13.014429 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:13.014401    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e3f789_5057_4149_adae_d626175767e0.slice/crio-4ff6b9ad74516d41cf9994b10e92b5018d10f7c28c9209fa6cebace1a828aa12 WatchSource:0}: Error finding container 4ff6b9ad74516d41cf9994b10e92b5018d10f7c28c9209fa6cebace1a828aa12: Status 404 returned error can't find the container with id 4ff6b9ad74516d41cf9994b10e92b5018d10f7c28c9209fa6cebace1a828aa12
Oct 29 23:00:13.016657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.016636    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"]
Oct 29 23:00:13.118126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.118103    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:13.431239 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:13.431186    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf146ab2e_c76c_46c7_8e0f_0379c50155da.slice/crio-962525734b448a2e1e809520142f7d0a75c30cb9aa39e7f95f08441c7686cf70 WatchSource:0}: Error finding container 962525734b448a2e1e809520142f7d0a75c30cb9aa39e7f95f08441c7686cf70: Status 404 returned error can't find the container with id 962525734b448a2e1e809520142f7d0a75c30cb9aa39e7f95f08441c7686cf70
Oct 29 23:00:13.432115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.432091    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"]
Oct 29 23:00:13.737918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.737881    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl" event={"ID":"bf0fa94f-5748-46f8-adeb-8ad243f3e72d","Type":"ContainerStarted","Data":"c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65"}
Oct 29 23:00:13.738069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.737950    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl" event={"ID":"bf0fa94f-5748-46f8-adeb-8ad243f3e72d","Type":"ContainerStarted","Data":"a8558edc71c8682974eb87691b5d36cbdade311ecb37a219550ee1dfab685ef9"}
Oct 29 23:00:13.738069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.737986    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:13.740044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.740015    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx" event={"ID":"f0e3f789-5057-4149-adae-d626175767e0","Type":"ContainerStarted","Data":"ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d"}
Oct 29 23:00:13.740176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.740067    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx" event={"ID":"f0e3f789-5057-4149-adae-d626175767e0","Type":"ContainerStarted","Data":"4ff6b9ad74516d41cf9994b10e92b5018d10f7c28c9209fa6cebace1a828aa12"}
Oct 29 23:00:13.740176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.740097    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:13.741555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.741531    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q" event={"ID":"f146ab2e-c76c-46c7-8e0f-0379c50155da","Type":"ContainerStarted","Data":"962525734b448a2e1e809520142f7d0a75c30cb9aa39e7f95f08441c7686cf70"}
Oct 29 23:00:13.780334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.780282    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl" podStartSLOduration=3.780269507 podStartE2EDuration="3.780269507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:13.778827327 +0000 UTC m=+1049.401738343" watchObservedRunningTime="2026-10-29 23:00:13.780269507 +0000 UTC m=+1049.403180527"
Oct 29 23:00:13.826101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:13.826055    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx" podStartSLOduration=3.826044141 podStartE2EDuration="3.826044141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:13.819929361 +0000 UTC m=+1049.442840381" watchObservedRunningTime="2026-10-29 23:00:13.826044141 +0000 UTC m=+1049.448955160"
Oct 29 23:00:14.057382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:14.057022    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"]
Oct 29 23:00:14.111499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:14.109339    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18711a78-lthrf"]
Oct 29 23:00:14.491098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:14.491067    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0b941f-a8d5-48ff-bb94-07afccec1815" path="/var/lib/kubelet/pods/cc0b941f-a8d5-48ff-bb94-07afccec1815/volumes"
Oct 29 23:00:14.754761 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:14.754670    4827 generic.go:356] "Generic (PLEG): container finished" podID="f146ab2e-c76c-46c7-8e0f-0379c50155da" containerID="e3441ef3f2a56bb8110bda2a1a74adff2ce5348aaf4165061cdabde34acec43a" exitCode=0
Oct 29 23:00:14.754989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:14.754813    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q" event={"ID":"f146ab2e-c76c-46c7-8e0f-0379c50155da","Type":"ContainerDied","Data":"e3441ef3f2a56bb8110bda2a1a74adff2ce5348aaf4165061cdabde34acec43a"}
Oct 29 23:00:15.490791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.483546    4827 scope.go:122] "RemoveContainer" containerID="80f8fbe0666278fbfb642fbc7e753ef03498e887b18601c933d24d373280a9b6"
Oct 29 23:00:15.619995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.619882    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"]
Oct 29 23:00:15.630012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.629960    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.641191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.641159    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"]
Oct 29 23:00:15.645780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.645733    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-13aa41-serving-cert\""
Oct 29 23:00:15.789081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.789049    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-metrics-certs\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.789331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.789313    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-maas-api-tls\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.789440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.789427    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a435022-535c-480e-b06b-d6da1508eb97-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.789494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.789482    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4crgj\" (UniqueName: \"kubernetes.io/projected/1a435022-535c-480e-b06b-d6da1508eb97-kube-api-access-4crgj\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.892237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.891812    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-metrics-certs\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.892237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.892138    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-maas-api-tls\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.892523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.892272    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a435022-535c-480e-b06b-d6da1508eb97-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.892523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.892321    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4crgj\" (UniqueName: \"kubernetes.io/projected/1a435022-535c-480e-b06b-d6da1508eb97-kube-api-access-4crgj\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.893258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.893233    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a435022-535c-480e-b06b-d6da1508eb97-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.895031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.895002    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-metrics-certs\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.895149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.895091    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-maas-api-tls\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:15.919811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:15.919119    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4crgj\" (UniqueName: \"kubernetes.io/projected/1a435022-535c-480e-b06b-d6da1508eb97-kube-api-access-4crgj\") pod \"maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:16.103872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.098109    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:16.244619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.244594    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:16.287990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.287016    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-openshift-ca\") pod \"f146ab2e-c76c-46c7-8e0f-0379c50155da\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") "
Oct 29 23:00:16.287990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.287451    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f146ab2e-c76c-46c7-8e0f-0379c50155da-kube-api-access-6x9rx\" (UniqueName: \"kubernetes.io/projected/f146ab2e-c76c-46c7-8e0f-0379c50155da-kube-api-access-6x9rx\") pod \"f146ab2e-c76c-46c7-8e0f-0379c50155da\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") "
Oct 29 23:00:16.287990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.287551    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-xks-ca\") pod \"f146ab2e-c76c-46c7-8e0f-0379c50155da\" (UID: \"f146ab2e-c76c-46c7-8e0f-0379c50155da\") "
Oct 29 23:00:16.289407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.289090    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-openshift-ca" pod "f146ab2e-c76c-46c7-8e0f-0379c50155da" (UID: "f146ab2e-c76c-46c7-8e0f-0379c50155da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:16.291503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.291481    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:16.299529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.299483    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-xks-ca" pod "f146ab2e-c76c-46c7-8e0f-0379c50155da" (UID: "f146ab2e-c76c-46c7-8e0f-0379c50155da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:16.530780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.520708    4827 scope.go:122] "RemoveContainer" containerID="399ff610ed52a2812e60197bfce6bef5bbe112f8e242df5bd7757d3a342eb920"
Oct 29 23:00:16.613164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.613084    4827 scope.go:122] "RemoveContainer" containerID="c55d40aafc029e0c4834e09ab4aa1ba4adcea4215d2e03d759a47c2d86dd8e68"
Oct 29 23:00:16.618990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.618956    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f146ab2e-c76c-46c7-8e0f-0379c50155da-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:16.638797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.636378    4827 scope.go:122] "RemoveContainer" containerID="b1847ad5c731435c3905eb36d32adb195be3c9eed02e14a984c5e1bac2b1f155"
Oct 29 23:00:16.639875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.639116    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f146ab2e-c76c-46c7-8e0f-0379c50155da-kube-api-access-6x9rx" pod "f146ab2e-c76c-46c7-8e0f-0379c50155da" (UID: "f146ab2e-c76c-46c7-8e0f-0379c50155da"). InnerVolumeSpecName "kube-api-access-6x9rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:16.727835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.721614    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9rx\" (UniqueName: \"kubernetes.io/projected/f146ab2e-c76c-46c7-8e0f-0379c50155da-kube-api-access-6x9rx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:16.809812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.809760    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q" event={"ID":"f146ab2e-c76c-46c7-8e0f-0379c50155da","Type":"ContainerDied","Data":"962525734b448a2e1e809520142f7d0a75c30cb9aa39e7f95f08441c7686cf70"}
Oct 29 23:00:16.809922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.809879    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="962525734b448a2e1e809520142f7d0a75c30cb9aa39e7f95f08441c7686cf70"
Oct 29 23:00:16.810310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.809991    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"
Oct 29 23:00:16.812800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:16.812758    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" event={"ID":"a6419ba1-4c1c-4a70-b654-183fa00dbd27","Type":"ContainerStarted","Data":"d568dd96406d2d0f05380c7f9383839ad71082425d2367585446403cc3e15fe7"}
Oct 29 23:00:17.006942 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:17.006910    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a435022_535c_480e_b06b_d6da1508eb97.slice/crio-9feb44314e869735982b840ad7e8a2b36b738fc723c61c6021c1e55ae08e0a8d WatchSource:0}: Error finding container 9feb44314e869735982b840ad7e8a2b36b738fc723c61c6021c1e55ae08e0a8d: Status 404 returned error can't find the container with id 9feb44314e869735982b840ad7e8a2b36b738fc723c61c6021c1e55ae08e0a8d
Oct 29 23:00:17.008317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:17.008293    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"]
Oct 29 23:00:17.540035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:17.514646    4827 scope.go:122] "RemoveContainer" containerID="91980bf633b8ae8e68aac8d5a2c6d7d226fe37f82c3bc07e5e2a1940a3fec74b"
Oct 29 23:00:17.540035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:17.515957    4827 scope.go:122] "RemoveContainer" containerID="c65876ef219ae1300337932e397e54582702b23166295bccb8cbe2f4a405abf0"
Oct 29 23:00:18.677882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:18.658061    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:18.916962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:18.909672    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:19.037006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.036981    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"]
Oct 29 23:00:19.037166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.037057    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"]
Oct 29 23:00:19.037166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.037088    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"]
Oct 29 23:00:19.156740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.156664    4827 generic.go:356] "Generic (PLEG): container finished" podID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerID="d568dd96406d2d0f05380c7f9383839ad71082425d2367585446403cc3e15fe7" exitCode=60
Oct 29 23:00:19.159195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.159147    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" event={"ID":"a6419ba1-4c1c-4a70-b654-183fa00dbd27","Type":"ContainerDied","Data":"d568dd96406d2d0f05380c7f9383839ad71082425d2367585446403cc3e15fe7"}
Oct 29 23:00:19.173444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.173409    4827 scope.go:122] "RemoveContainer" containerID="80f8fbe0666278fbfb642fbc7e753ef03498e887b18601c933d24d373280a9b6"
Oct 29 23:00:19.342402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.342369    4827 generic.go:356] "Generic (PLEG): container finished" podID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerID="a0420fef29f16ea9a567ed0e191edd775d2741f83f8b94291269cd462d7515a5" exitCode=0
Oct 29 23:00:19.342732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.342700    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" event={"ID":"c197150f-64b3-41b9-a0ed-951e3cb3212d","Type":"ContainerDied","Data":"a0420fef29f16ea9a567ed0e191edd775d2741f83f8b94291269cd462d7515a5"}
Oct 29 23:00:19.499483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.499431    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 23:00:19.503391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.503361    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx" event={"ID":"1a435022-535c-480e-b06b-d6da1508eb97","Type":"ContainerStarted","Data":"9feb44314e869735982b840ad7e8a2b36b738fc723c61c6021c1e55ae08e0a8d"}
Oct 29 23:00:19.650857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.623551    4827 scope.go:122] "RemoveContainer" containerID="4f62da5ddd3f9310faa1c140ef75a3f5e4e43704bfb3ddbcf922bef4882e1c5d"
Oct 29 23:00:19.940967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.933719    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c197150f-64b3-41b9-a0ed-951e3cb3212d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c197150f-64b3-41b9-a0ed-951e3cb3212d-istiod-ca-cert\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.941548 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.941522    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-data\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.941824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.941806    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-certs\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.941989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.941974    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-credential-socket\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.942149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.942136    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-podinfo\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.942270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.942258    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-socket\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.942410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.942398    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-kube-api-access-gh9th\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-kube-api-access-gh9th\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.942538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.942522    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-envoy\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.942679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.942668    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-token\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-token\") pod \"c197150f-64b3-41b9-a0ed-951e3cb3212d\" (UID: \"c197150f-64b3-41b9-a0ed-951e3cb3212d\") "
Oct 29 23:00:19.945535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.945504    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c197150f-64b3-41b9-a0ed-951e3cb3212d-istiod-ca-cert" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:19.946069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.946046    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-data" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:19.946799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.946778    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-certs" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:19.992657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.991949    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-podinfo" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:19.992657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:19.992302    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-socket" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:20.052022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.041400    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-credential-socket" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:20.182913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.182871    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-kube-api-access-gh9th" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "kube-api-access-gh9th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.197699    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-token" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.198097    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-envoy" pod "c197150f-64b3-41b9-a0ed-951e3cb3212d" (UID: "c197150f-64b3-41b9-a0ed-951e3cb3212d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202755    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202828    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202844    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202859    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202874    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9th\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-kube-api-access-gh9th\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202888    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202900    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202913    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c197150f-64b3-41b9-a0ed-951e3cb3212d-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.204674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.202926    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c197150f-64b3-41b9-a0ed-951e3cb3212d-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.230676 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.230651    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:20.303709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.303676    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6419ba1-4c1c-4a70-b654-183fa00dbd27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6419ba1-4c1c-4a70-b654-183fa00dbd27-maas-api-service-ca\") pod \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") "
Oct 29 23:00:20.303935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.303916    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6419ba1-4c1c-4a70-b654-183fa00dbd27-kube-api-access-bfqrr\" (UniqueName: \"kubernetes.io/projected/a6419ba1-4c1c-4a70-b654-183fa00dbd27-kube-api-access-bfqrr\") pod \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\" (UID: \"a6419ba1-4c1c-4a70-b654-183fa00dbd27\") "
Oct 29 23:00:20.304145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.304115    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6419ba1-4c1c-4a70-b654-183fa00dbd27-maas-api-service-ca" pod "a6419ba1-4c1c-4a70-b654-183fa00dbd27" (UID: "a6419ba1-4c1c-4a70-b654-183fa00dbd27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.304838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.304812    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6419ba1-4c1c-4a70-b654-183fa00dbd27-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.306275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.306249    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6419ba1-4c1c-4a70-b654-183fa00dbd27-kube-api-access-bfqrr" pod "a6419ba1-4c1c-4a70-b654-183fa00dbd27" (UID: "a6419ba1-4c1c-4a70-b654-183fa00dbd27"). InnerVolumeSpecName "kube-api-access-bfqrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.406857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.406824    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqrr\" (UniqueName: \"kubernetes.io/projected/a6419ba1-4c1c-4a70-b654-183fa00dbd27-kube-api-access-bfqrr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.513651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.513621    4827 generic.go:356] "Generic (PLEG): container finished" podID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerID="83d72d4bd513080586712954b541092a75a1bdff12c263795c54c3e473e18b9e" exitCode=60
Oct 29 23:00:20.513802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.513686    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" event={"ID":"cbe05cd7-9c9d-455a-8a8b-69c903af653e","Type":"ContainerDied","Data":"83d72d4bd513080586712954b541092a75a1bdff12c263795c54c3e473e18b9e"}
Oct 29 23:00:20.513802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.513736    4827 scope.go:122] "RemoveContainer" containerID="b1847ad5c731435c3905eb36d32adb195be3c9eed02e14a984c5e1bac2b1f155"
Oct 29 23:00:20.514074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.514059    4827 scope.go:122] "RemoveContainer" containerID="83d72d4bd513080586712954b541092a75a1bdff12c263795c54c3e473e18b9e"
Oct 29 23:00:20.514350 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:20.514326    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t_odh-ai-gateway-infra(cbe05cd7-9c9d-455a-8a8b-69c903af653e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e"
Oct 29 23:00:20.516026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.516001    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" event={"ID":"cd33576d-8fab-40c9-8690-0f770df77547","Type":"ContainerStarted","Data":"fd3aeb9f40d59eb1b380adf220ea866fa1839811939fc14d74ac78a34909704a"}
Oct 29 23:00:20.518152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.518131    4827 generic.go:356] "Generic (PLEG): container finished" podID="117c6362-84ed-4158-a670-9404fa625943" containerID="d6d2fb467e3d7130c590787b31df9327a75e7d61d7349ca2631693b8097937cd" exitCode=60
Oct 29 23:00:20.518242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.518198    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" event={"ID":"117c6362-84ed-4158-a670-9404fa625943","Type":"ContainerDied","Data":"d6d2fb467e3d7130c590787b31df9327a75e7d61d7349ca2631693b8097937cd"}
Oct 29 23:00:20.518515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.518501    4827 scope.go:122] "RemoveContainer" containerID="d6d2fb467e3d7130c590787b31df9327a75e7d61d7349ca2631693b8097937cd"
Oct 29 23:00:20.518826 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:20.518807    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl_odh-ai-gateway-infra(117c6362-84ed-4158-a670-9404fa625943)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" podUID="117c6362-84ed-4158-a670-9404fa625943"
Oct 29 23:00:20.520452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.520431    4827 generic.go:356] "Generic (PLEG): container finished" podID="273ae80a-cc37-4511-9e1a-485554e20f43" containerID="8af97200565d09d99fb43686c73cf721415f15feeba7b0c263b59c8e02da6f8d" exitCode=60
Oct 29 23:00:20.520546 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.520471    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" event={"ID":"273ae80a-cc37-4511-9e1a-485554e20f43","Type":"ContainerDied","Data":"8af97200565d09d99fb43686c73cf721415f15feeba7b0c263b59c8e02da6f8d"}
Oct 29 23:00:20.520905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.520887    4827 scope.go:122] "RemoveContainer" containerID="8af97200565d09d99fb43686c73cf721415f15feeba7b0c263b59c8e02da6f8d"
Oct 29 23:00:20.521231 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:20.521199    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs_odh-ai-gateway-infra(273ae80a-cc37-4511-9e1a-485554e20f43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" podUID="273ae80a-cc37-4511-9e1a-485554e20f43"
Oct 29 23:00:20.522124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.522107    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"
Oct 29 23:00:20.522498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.522471    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8" event={"ID":"a6419ba1-4c1c-4a70-b654-183fa00dbd27","Type":"ContainerDied","Data":"473065d8f02302e4427122188ad330b626bd5a40d9a7aacf3a2d8207b9610866"}
Oct 29 23:00:20.523717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.523697    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2" event={"ID":"c197150f-64b3-41b9-a0ed-951e3cb3212d","Type":"ContainerDied","Data":"6e7fbc85e45f8845a74c8aa99ccd759f3338ac3034428546a6958cf665f08447"}
Oct 29 23:00:20.523834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.523706    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"
Oct 29 23:00:20.525988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.525966    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" event={"ID":"fece158c-1f91-4810-9c88-df9aaf8ef8be","Type":"ContainerStarted","Data":"f47947508ef8e4777614fdfc8c5e7336fb25362c47a4897e0a8811d5214837d9"}
Oct 29 23:00:20.528303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.528281    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx" event={"ID":"1a435022-535c-480e-b06b-d6da1508eb97","Type":"ContainerStarted","Data":"c8012efaed1ef503bdd1d66320bca8e6403d95ce168f7ce7ea91c4a49e5efcd6"}
Oct 29 23:00:20.528577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.528332    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:20.530588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.530565    4827 scope.go:122] "RemoveContainer" containerID="c55d40aafc029e0c4834e09ab4aa1ba4adcea4215d2e03d759a47c2d86dd8e68"
Oct 29 23:00:20.544699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.544682    4827 scope.go:122] "RemoveContainer" containerID="399ff610ed52a2812e60197bfce6bef5bbe112f8e242df5bd7757d3a342eb920"
Oct 29 23:00:20.559174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.559156    4827 scope.go:122] "RemoveContainer" containerID="d568dd96406d2d0f05380c7f9383839ad71082425d2367585446403cc3e15fe7"
Oct 29 23:00:20.571301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.571284    4827 scope.go:122] "RemoveContainer" containerID="a0420fef29f16ea9a567ed0e191edd775d2741f83f8b94291269cd462d7515a5"
Oct 29 23:00:20.626420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.626386    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"]
Oct 29 23:00:20.634010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.633978    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-zzdd2"]
Oct 29 23:00:20.667107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.667077    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"]
Oct 29 23:00:20.673732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.673703    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888580-9f7c8"]
Oct 29 23:00:20.829988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:20.829893    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx" podStartSLOduration=3.814651316 podStartE2EDuration="5.829878669s" totalImagesPullingTime="2.015227353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:20.824428936 +0000 UTC m=+1056.447339982" watchObservedRunningTime="2026-10-29 23:00:20.829878669 +0000 UTC m=+1056.452789688"
Oct 29 23:00:21.544866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.544834    4827 generic.go:356] "Generic (PLEG): container finished" podID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerID="f47947508ef8e4777614fdfc8c5e7336fb25362c47a4897e0a8811d5214837d9" exitCode=60
Oct 29 23:00:21.545296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.544897    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" event={"ID":"fece158c-1f91-4810-9c88-df9aaf8ef8be","Type":"ContainerDied","Data":"f47947508ef8e4777614fdfc8c5e7336fb25362c47a4897e0a8811d5214837d9"}
Oct 29 23:00:21.545296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.544953    4827 scope.go:122] "RemoveContainer" containerID="c65876ef219ae1300337932e397e54582702b23166295bccb8cbe2f4a405abf0"
Oct 29 23:00:21.545404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.545329    4827 scope.go:122] "RemoveContainer" containerID="f47947508ef8e4777614fdfc8c5e7336fb25362c47a4897e0a8811d5214837d9"
Oct 29 23:00:21.545702 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:21.545673    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j_odh-ai-gateway-infra(fece158c-1f91-4810-9c88-df9aaf8ef8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be"
Oct 29 23:00:21.548697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.548678    4827 generic.go:356] "Generic (PLEG): container finished" podID="cd33576d-8fab-40c9-8690-0f770df77547" containerID="fd3aeb9f40d59eb1b380adf220ea866fa1839811939fc14d74ac78a34909704a" exitCode=60
Oct 29 23:00:21.548854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.548735    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" event={"ID":"cd33576d-8fab-40c9-8690-0f770df77547","Type":"ContainerDied","Data":"fd3aeb9f40d59eb1b380adf220ea866fa1839811939fc14d74ac78a34909704a"}
Oct 29 23:00:21.549183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.549168    4827 scope.go:122] "RemoveContainer" containerID="fd3aeb9f40d59eb1b380adf220ea866fa1839811939fc14d74ac78a34909704a"
Oct 29 23:00:21.549490 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:21.549469    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m_odh-ai-gateway-infra(cd33576d-8fab-40c9-8690-0f770df77547)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" podUID="cd33576d-8fab-40c9-8690-0f770df77547"
Oct 29 23:00:21.560118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.560102    4827 scope.go:122] "RemoveContainer" containerID="91980bf633b8ae8e68aac8d5a2c6d7d226fe37f82c3bc07e5e2a1940a3fec74b"
Oct 29 23:00:21.567233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.567212    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"]
Oct 29 23:00:21.724000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.723968    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"]
Oct 29 23:00:21.756953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.756922    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"]
Oct 29 23:00:21.765397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.765376    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:21.798210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.798138    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"]
Oct 29 23:00:21.838790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.838728    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe05cd7-9c9d-455a-8a8b-69c903af653e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe05cd7-9c9d-455a-8a8b-69c903af653e-maas-api-service-ca\") pod \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") "
Oct 29 23:00:21.838905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.838802    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe05cd7-9c9d-455a-8a8b-69c903af653e-kube-api-access-8rh79\" (UniqueName: \"kubernetes.io/projected/cbe05cd7-9c9d-455a-8a8b-69c903af653e-kube-api-access-8rh79\") pod \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\" (UID: \"cbe05cd7-9c9d-455a-8a8b-69c903af653e\") "
Oct 29 23:00:21.841621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.841524    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe05cd7-9c9d-455a-8a8b-69c903af653e-maas-api-service-ca" pod "cbe05cd7-9c9d-455a-8a8b-69c903af653e" (UID: "cbe05cd7-9c9d-455a-8a8b-69c903af653e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:21.882798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.866136    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe05cd7-9c9d-455a-8a8b-69c903af653e-kube-api-access-8rh79" pod "cbe05cd7-9c9d-455a-8a8b-69c903af653e" (UID: "cbe05cd7-9c9d-455a-8a8b-69c903af653e"). InnerVolumeSpecName "kube-api-access-8rh79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:21.882798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.882542    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"]
Oct 29 23:00:21.949761 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.949735    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe05cd7-9c9d-455a-8a8b-69c903af653e-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.949761 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.949783    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rh79\" (UniqueName: \"kubernetes.io/projected/cbe05cd7-9c9d-455a-8a8b-69c903af653e-kube-api-access-8rh79\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.969748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:21.969724    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:22.037062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.037034    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:22.050496 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.050425    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117c6362-84ed-4158-a670-9404fa625943-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117c6362-84ed-4158-a670-9404fa625943-maas-api-service-ca\") pod \"117c6362-84ed-4158-a670-9404fa625943\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") "
Oct 29 23:00:22.050651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.050627    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117c6362-84ed-4158-a670-9404fa625943-kube-api-access-cww5j\" (UniqueName: \"kubernetes.io/projected/117c6362-84ed-4158-a670-9404fa625943-kube-api-access-cww5j\") pod \"117c6362-84ed-4158-a670-9404fa625943\" (UID: \"117c6362-84ed-4158-a670-9404fa625943\") "
Oct 29 23:00:22.050849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.050827    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117c6362-84ed-4158-a670-9404fa625943-maas-api-service-ca" pod "117c6362-84ed-4158-a670-9404fa625943" (UID: "117c6362-84ed-4158-a670-9404fa625943"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.051388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.051368    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117c6362-84ed-4158-a670-9404fa625943-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.053616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.053534    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117c6362-84ed-4158-a670-9404fa625943-kube-api-access-cww5j" pod "117c6362-84ed-4158-a670-9404fa625943" (UID: "117c6362-84ed-4158-a670-9404fa625943"). InnerVolumeSpecName "kube-api-access-cww5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.152735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.152698    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273ae80a-cc37-4511-9e1a-485554e20f43-kube-api-access-wgsk5\" (UniqueName: \"kubernetes.io/projected/273ae80a-cc37-4511-9e1a-485554e20f43-kube-api-access-wgsk5\") pod \"273ae80a-cc37-4511-9e1a-485554e20f43\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") "
Oct 29 23:00:22.153517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.153490    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273ae80a-cc37-4511-9e1a-485554e20f43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ae80a-cc37-4511-9e1a-485554e20f43-maas-api-service-ca\") pod \"273ae80a-cc37-4511-9e1a-485554e20f43\" (UID: \"273ae80a-cc37-4511-9e1a-485554e20f43\") "
Oct 29 23:00:22.178888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.178854    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cww5j\" (UniqueName: \"kubernetes.io/projected/117c6362-84ed-4158-a670-9404fa625943-kube-api-access-cww5j\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.180406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.175077    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273ae80a-cc37-4511-9e1a-485554e20f43-maas-api-service-ca" pod "273ae80a-cc37-4511-9e1a-485554e20f43" (UID: "273ae80a-cc37-4511-9e1a-485554e20f43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.180891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.180857    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273ae80a-cc37-4511-9e1a-485554e20f43-kube-api-access-wgsk5" pod "273ae80a-cc37-4511-9e1a-485554e20f43" (UID: "273ae80a-cc37-4511-9e1a-485554e20f43"). InnerVolumeSpecName "kube-api-access-wgsk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.299788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.294403    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsk5\" (UniqueName: \"kubernetes.io/projected/273ae80a-cc37-4511-9e1a-485554e20f43-kube-api-access-wgsk5\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.299788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.294439    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/273ae80a-cc37-4511-9e1a-485554e20f43-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.492100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.492042    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" path="/var/lib/kubelet/pods/a6419ba1-4c1c-4a70-b654-183fa00dbd27/volumes"
Oct 29 23:00:22.493147 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.493115    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" path="/var/lib/kubelet/pods/c197150f-64b3-41b9-a0ed-951e3cb3212d/volumes"
Oct 29 23:00:22.562413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.562382    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"
Oct 29 23:00:22.562860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.562379    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t" event={"ID":"cbe05cd7-9c9d-455a-8a8b-69c903af653e","Type":"ContainerDied","Data":"08ff8da91ec5119eedfd1809d00ee764cc75954c316df3dfed54be2d08f8ff75"}
Oct 29 23:00:22.562860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.562523    4827 scope.go:122] "RemoveContainer" containerID="83d72d4bd513080586712954b541092a75a1bdff12c263795c54c3e473e18b9e"
Oct 29 23:00:22.566081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.566064    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"
Oct 29 23:00:22.566193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.566071    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl" event={"ID":"117c6362-84ed-4158-a670-9404fa625943","Type":"ContainerDied","Data":"97507001b44cf3f66727f869d41914b625a2a5211c39b51d9a2388eb27f7bca1"}
Oct 29 23:00:22.569145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.569121    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs" event={"ID":"273ae80a-cc37-4511-9e1a-485554e20f43","Type":"ContainerDied","Data":"e02b3a1763bbb3906ca74ae2adc65813fb9f1884e4332815e681417b0f9830b8"}
Oct 29 23:00:22.569346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.569332    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"
Oct 29 23:00:22.581685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.581643    4827 scope.go:122] "RemoveContainer" containerID="d6d2fb467e3d7130c590787b31df9327a75e7d61d7349ca2631693b8097937cd"
Oct 29 23:00:22.596636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.596619    4827 scope.go:122] "RemoveContainer" containerID="8af97200565d09d99fb43686c73cf721415f15feeba7b0c263b59c8e02da6f8d"
Oct 29 23:00:22.666686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.666648    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"]
Oct 29 23:00:22.675868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.675834    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888580-qsjfs"]
Oct 29 23:00:22.717363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.716691    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"]
Oct 29 23:00:22.726502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.726477    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888580-nbjhl"]
Oct 29 23:00:22.766468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.766436    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:22.786629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.786599    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"]
Oct 29 23:00:22.795623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.795596    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888580-xsq2t"]
Oct 29 23:00:22.805461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.805438    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:22.922090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.922053    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd33576d-8fab-40c9-8690-0f770df77547-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd33576d-8fab-40c9-8690-0f770df77547-maas-api-service-ca\") pod \"cd33576d-8fab-40c9-8690-0f770df77547\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") "
Oct 29 23:00:22.922243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.922166    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fece158c-1f91-4810-9c88-df9aaf8ef8be-kube-api-access-cj485\" (UniqueName: \"kubernetes.io/projected/fece158c-1f91-4810-9c88-df9aaf8ef8be-kube-api-access-cj485\") pod \"fece158c-1f91-4810-9c88-df9aaf8ef8be\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") "
Oct 29 23:00:22.922351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.922336    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd33576d-8fab-40c9-8690-0f770df77547-kube-api-access-7h2kf\" (UniqueName: \"kubernetes.io/projected/cd33576d-8fab-40c9-8690-0f770df77547-kube-api-access-7h2kf\") pod \"cd33576d-8fab-40c9-8690-0f770df77547\" (UID: \"cd33576d-8fab-40c9-8690-0f770df77547\") "
Oct 29 23:00:22.922466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.922446    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fece158c-1f91-4810-9c88-df9aaf8ef8be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece158c-1f91-4810-9c88-df9aaf8ef8be-maas-api-service-ca\") pod \"fece158c-1f91-4810-9c88-df9aaf8ef8be\" (UID: \"fece158c-1f91-4810-9c88-df9aaf8ef8be\") "
Oct 29 23:00:22.922536 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.922473    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd33576d-8fab-40c9-8690-0f770df77547-maas-api-service-ca" pod "cd33576d-8fab-40c9-8690-0f770df77547" (UID: "cd33576d-8fab-40c9-8690-0f770df77547"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.923230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.923206    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd33576d-8fab-40c9-8690-0f770df77547-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.923347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.923286    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fece158c-1f91-4810-9c88-df9aaf8ef8be-maas-api-service-ca" pod "fece158c-1f91-4810-9c88-df9aaf8ef8be" (UID: "fece158c-1f91-4810-9c88-df9aaf8ef8be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.927000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.926974    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd33576d-8fab-40c9-8690-0f770df77547-kube-api-access-7h2kf" pod "cd33576d-8fab-40c9-8690-0f770df77547" (UID: "cd33576d-8fab-40c9-8690-0f770df77547"). InnerVolumeSpecName "kube-api-access-7h2kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.927110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:22.927018    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fece158c-1f91-4810-9c88-df9aaf8ef8be-kube-api-access-cj485" pod "fece158c-1f91-4810-9c88-df9aaf8ef8be" (UID: "fece158c-1f91-4810-9c88-df9aaf8ef8be"). InnerVolumeSpecName "kube-api-access-cj485". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:23.025244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.025172    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj485\" (UniqueName: \"kubernetes.io/projected/fece158c-1f91-4810-9c88-df9aaf8ef8be-kube-api-access-cj485\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.025244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.025198    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2kf\" (UniqueName: \"kubernetes.io/projected/cd33576d-8fab-40c9-8690-0f770df77547-kube-api-access-7h2kf\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.025244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.025208    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece158c-1f91-4810-9c88-df9aaf8ef8be-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.039510 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.039480    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"]
Oct 29 23:00:23.045776 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.045740    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd27b89-8jb2r"]
Oct 29 23:00:23.583670 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.583642    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"
Oct 29 23:00:23.584161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.583686    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j" event={"ID":"fece158c-1f91-4810-9c88-df9aaf8ef8be","Type":"ContainerDied","Data":"500aa67eddc369e83545a553c4d17b5d297e9592a1b9c0920494db626c1446eb"}
Oct 29 23:00:23.584161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.583747    4827 scope.go:122] "RemoveContainer" containerID="f47947508ef8e4777614fdfc8c5e7336fb25362c47a4897e0a8811d5214837d9"
Oct 29 23:00:23.591917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.591890    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m" event={"ID":"cd33576d-8fab-40c9-8690-0f770df77547","Type":"ContainerDied","Data":"f7fc0af8b2983b30fc8a367c25e0df356039336f1b9df97fcf0330339913b166"}
Oct 29 23:00:23.591917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.591904    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"
Oct 29 23:00:23.599945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.599923    4827 scope.go:122] "RemoveContainer" containerID="fd3aeb9f40d59eb1b380adf220ea866fa1839811939fc14d74ac78a34909704a"
Oct 29 23:00:23.641832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.641791    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"]
Oct 29 23:00:23.647022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.646998    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"]
Oct 29 23:00:23.647803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647786    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647803    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647823    4827 cpu_manager.go:414] "containerMap: removing container" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647829    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647838    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647843    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647851    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647855    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647865    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647869    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647877    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647881    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647889    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.647895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647896    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647911    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647917    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647928    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647932    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647939    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647944    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647956    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647960    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647970    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647977    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647992    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.647996    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648005    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f146ab2e-c76c-46c7-8e0f-0379c50155da" containerName="revoke-keys"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648009    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f146ab2e-c76c-46c7-8e0f-0379c50155da" containerName="revoke-keys"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648017    4827 cpu_manager.go:414] "containerMap: removing container" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648022    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648033    4827 cpu_manager.go:414] "containerMap: removing container" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648038    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648049    4827 cpu_manager.go:414] "containerMap: removing container" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648055    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648067    4827 cpu_manager.go:414] "containerMap: removing container" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648072    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648191    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648200    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648206    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648218    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648227    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648232    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648237    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648242    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648247    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648251    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6419ba1-4c1c-4a70-b654-183fa00dbd27" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648258    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648263    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648270    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c197150f-64b3-41b9-a0ed-951e3cb3212d" containerName="istio-proxy"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648276    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648281    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648290    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:23.648488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.648299    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f146ab2e-c76c-46c7-8e0f-0379c50155da" containerName="revoke-keys"
Oct 29 23:00:23.653651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.653635    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:23.660971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.660950    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3092a4d-29888580-scq2j"]
Oct 29 23:00:23.671565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.671517    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-774bff\""
Oct 29 23:00:23.672630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.672497    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"]
Oct 29 23:00:23.688350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.688307    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-774bff-dockercfg-kl8t7\""
Oct 29 23:00:23.737123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.737091    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25899eb6-4369-419b-8485-ce80c082afd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:23.737401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.737279    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggd8m\" (UniqueName: \"kubernetes.io/projected/25899eb6-4369-419b-8485-ce80c082afd0-kube-api-access-ggd8m\") pod \"payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:23.754316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.754284    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"]
Oct 29 23:00:23.760665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.760367    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888580-lkf4m"]
Oct 29 23:00:23.873852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.859055    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25899eb6-4369-419b-8485-ce80c082afd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:23.873852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.859242    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggd8m\" (UniqueName: \"kubernetes.io/projected/25899eb6-4369-419b-8485-ce80c082afd0-kube-api-access-ggd8m\") pod \"payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:23.873852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:23.860704    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25899eb6-4369-419b-8485-ce80c082afd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:24.046716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.046370    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl" podUID="8c28df07-1331-4dc8-90dc-43d93c8ddae1" containerName="payload-processing" containerID="cri-o://7cb8b8e24118a98fdde5fce551d4d0239538f97091ab08bc7a571221b0115c92" gracePeriod=25
Oct 29 23:00:24.046716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.046600    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd" podUID="28287833-a7b4-440c-b3e4-04c369bd615c" containerName="payload-pre-processing" containerID="cri-o://0c94d7fdfc70da120d766dca91eaf8387b092a2b6aaf32f5c5353da089f661da" gracePeriod=25
Oct 29 23:00:24.109332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.078231    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"]
Oct 29 23:00:24.109332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.084642    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggd8m\" (UniqueName: \"kubernetes.io/projected/25899eb6-4369-419b-8485-ce80c082afd0-kube-api-access-ggd8m\") pod \"payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:24.301901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.301865    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:24.432835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.427284    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"]
Oct 29 23:00:24.737563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.737478    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117c6362-84ed-4158-a670-9404fa625943" path="/var/lib/kubelet/pods/117c6362-84ed-4158-a670-9404fa625943/volumes"
Oct 29 23:00:24.759398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.759370    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" path="/var/lib/kubelet/pods/273ae80a-cc37-4511-9e1a-485554e20f43/volumes"
Oct 29 23:00:24.798154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.798106    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" path="/var/lib/kubelet/pods/cbe05cd7-9c9d-455a-8a8b-69c903af653e/volumes"
Oct 29 23:00:24.804648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.804181    4827 generic.go:356] "Generic (PLEG): container finished" podID="8c28df07-1331-4dc8-90dc-43d93c8ddae1" containerID="7cb8b8e24118a98fdde5fce551d4d0239538f97091ab08bc7a571221b0115c92" exitCode=0
Oct 29 23:00:24.806789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.806562    4827 generic.go:356] "Generic (PLEG): container finished" podID="28287833-a7b4-440c-b3e4-04c369bd615c" containerID="0c94d7fdfc70da120d766dca91eaf8387b092a2b6aaf32f5c5353da089f661da" exitCode=0
Oct 29 23:00:24.809164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.809137    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd33576d-8fab-40c9-8690-0f770df77547" path="/var/lib/kubelet/pods/cd33576d-8fab-40c9-8690-0f770df77547/volumes"
Oct 29 23:00:24.810075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.810054    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" path="/var/lib/kubelet/pods/fece158c-1f91-4810-9c88-df9aaf8ef8be/volumes"
Oct 29 23:00:24.811608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.811590    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd" path="/var/lib/kubelet/pods/ffb5f503-a4ea-4bc8-9d53-ee3905bdd4bd/volumes"
Oct 29 23:00:24.812371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.812356    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:00:24.812570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.812558    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:00:24.812699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.812683    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl" event={"ID":"8c28df07-1331-4dc8-90dc-43d93c8ddae1","Type":"ContainerDied","Data":"7cb8b8e24118a98fdde5fce551d4d0239538f97091ab08bc7a571221b0115c92"}
Oct 29 23:00:24.812881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:24.812864    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd" event={"ID":"28287833-a7b4-440c-b3e4-04c369bd615c","Type":"ContainerDied","Data":"0c94d7fdfc70da120d766dca91eaf8387b092a2b6aaf32f5c5353da089f661da"}
Oct 29 23:00:25.350052 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:25.349798    4827 logging.go:55] [core] [Channel #1748 SubChannel #1749]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.135:50052: connect: connection refused"
Oct 29 23:00:25.362599 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:25.360871    4827 logging.go:55] [core] [Channel #1750 SubChannel #1751]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.134:50052", ServerName: "10.134.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.134:50052: connect: connection refused"
Oct 29 23:00:25.654803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.647856    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"]
Oct 29 23:00:25.699079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.699055    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 23:00:25.727876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.727851    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c28df07-1331-4dc8-90dc-43d93c8ddae1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28df07-1331-4dc8-90dc-43d93c8ddae1-plugins-config-volume\") pod \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") "
Oct 29 23:00:25.728173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.728157    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c28df07-1331-4dc8-90dc-43d93c8ddae1-kube-api-access-4tpc6\" (UniqueName: \"kubernetes.io/projected/8c28df07-1331-4dc8-90dc-43d93c8ddae1-kube-api-access-4tpc6\") pod \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\" (UID: \"8c28df07-1331-4dc8-90dc-43d93c8ddae1\") "
Oct 29 23:00:25.728245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.728183    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c28df07-1331-4dc8-90dc-43d93c8ddae1-plugins-config-volume" pod "8c28df07-1331-4dc8-90dc-43d93c8ddae1" (UID: "8c28df07-1331-4dc8-90dc-43d93c8ddae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:25.728876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.728854    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28df07-1331-4dc8-90dc-43d93c8ddae1-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.730371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.730339    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c28df07-1331-4dc8-90dc-43d93c8ddae1-kube-api-access-4tpc6" pod "8c28df07-1331-4dc8-90dc-43d93c8ddae1" (UID: "8c28df07-1331-4dc8-90dc-43d93c8ddae1"). InnerVolumeSpecName "kube-api-access-4tpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:25.760700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.760673    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 23:00:25.819004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.818979    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"
Oct 29 23:00:25.819279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.819011    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl" event={"ID":"8c28df07-1331-4dc8-90dc-43d93c8ddae1","Type":"ContainerDied","Data":"1a1d315fc451ecf17b8ef198068a3f9c0ce206a9199320c2eeab4aecec16a8fa"}
Oct 29 23:00:25.819471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.819333    4827 scope.go:122] "RemoveContainer" containerID="7cb8b8e24118a98fdde5fce551d4d0239538f97091ab08bc7a571221b0115c92"
Oct 29 23:00:25.821026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.821001    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd" event={"ID":"28287833-a7b4-440c-b3e4-04c369bd615c","Type":"ContainerDied","Data":"aed27c2d2523444a8a26e4e04456d43c6162a1472ee6400d6ae29b67b69629fb"}
Oct 29 23:00:25.821171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.821087    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"
Oct 29 23:00:25.823631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.823608    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq" event={"ID":"25899eb6-4369-419b-8485-ce80c082afd0","Type":"ContainerStarted","Data":"4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78"}
Oct 29 23:00:25.823740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.823644    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq" event={"ID":"25899eb6-4369-419b-8485-ce80c082afd0","Type":"ContainerStarted","Data":"bb36a8fa455864d8a84e9d631b11ef32c768ad6c082bba0fcf577aaf19b856cd"}
Oct 29 23:00:25.824009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.823970    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:25.830335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.830306    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28287833-a7b4-440c-b3e4-04c369bd615c-kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/28287833-a7b4-440c-b3e4-04c369bd615c-kube-api-access-xq9bg\") pod \"28287833-a7b4-440c-b3e4-04c369bd615c\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") "
Oct 29 23:00:25.830445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.830362    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28287833-a7b4-440c-b3e4-04c369bd615c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28287833-a7b4-440c-b3e4-04c369bd615c-plugins-config-volume\") pod \"28287833-a7b4-440c-b3e4-04c369bd615c\" (UID: \"28287833-a7b4-440c-b3e4-04c369bd615c\") "
Oct 29 23:00:25.831146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.831110    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28287833-a7b4-440c-b3e4-04c369bd615c-plugins-config-volume" pod "28287833-a7b4-440c-b3e4-04c369bd615c" (UID: "28287833-a7b4-440c-b3e4-04c369bd615c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:25.831233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.831169    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpc6\" (UniqueName: \"kubernetes.io/projected/8c28df07-1331-4dc8-90dc-43d93c8ddae1-kube-api-access-4tpc6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.832758 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.832729    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28287833-a7b4-440c-b3e4-04c369bd615c-kube-api-access-xq9bg" pod "28287833-a7b4-440c-b3e4-04c369bd615c" (UID: "28287833-a7b4-440c-b3e4-04c369bd615c"). InnerVolumeSpecName "kube-api-access-xq9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:25.835546 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.835526    4827 scope.go:122] "RemoveContainer" containerID="0c94d7fdfc70da120d766dca91eaf8387b092a2b6aaf32f5c5353da089f661da"
Oct 29 23:00:25.876257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.876226    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"]
Oct 29 23:00:25.882663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.882644    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl"]
Oct 29 23:00:25.932032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.931978    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq" podStartSLOduration=2.9319609250000003 podStartE2EDuration="2.931960925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:25.92813374 +0000 UTC m=+1061.551044760" watchObservedRunningTime="2026-10-29 23:00:25.931960925 +0000 UTC m=+1061.554871944"
Oct 29 23:00:25.933085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.933059    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/28287833-a7b4-440c-b3e4-04c369bd615c-kube-api-access-xq9bg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.933085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:25.933083    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28287833-a7b4-440c-b3e4-04c369bd615c-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:26.032947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.032915    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"]
Oct 29 23:00:26.040327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.040293    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91a00f4-bvt4q"]
Oct 29 23:00:26.362591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.362542    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-gq2pl" podUID="8c28df07-1331-4dc8-90dc-43d93c8ddae1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded"
Oct 29 23:00:26.363245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.363212    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd" podUID="28287833-a7b4-440c-b3e4-04c369bd615c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.134:50052\" within 1s: context deadline exceeded"
Oct 29 23:00:26.378441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.378406    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"]
Oct 29 23:00:26.385574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.385521    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-v4thd"]
Oct 29 23:00:26.494246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.494211    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28287833-a7b4-440c-b3e4-04c369bd615c" path="/var/lib/kubelet/pods/28287833-a7b4-440c-b3e4-04c369bd615c/volumes"
Oct 29 23:00:26.494754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.494740    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c28df07-1331-4dc8-90dc-43d93c8ddae1" path="/var/lib/kubelet/pods/8c28df07-1331-4dc8-90dc-43d93c8ddae1/volumes"
Oct 29 23:00:26.495228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.495214    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8754b19-1f30-4e9e-900f-cd099ed18acf" path="/var/lib/kubelet/pods/c8754b19-1f30-4e9e-900f-cd099ed18acf/volumes"
Oct 29 23:00:26.556535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:26.556460    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:00:27.853787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:27.852312    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"]
Oct 29 23:00:27.853787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:27.852568    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6" podUID="d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" containerName="maas-api" containerID="cri-o://7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135" gracePeriod=30
Oct 29 23:00:28.447971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:28.447936    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"]
Oct 29 23:00:28.589684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:28.580174    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a0dbf5-94mfd"]
Oct 29 23:00:28.845303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:28.827182    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6" podUID="d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.138:8443/health\": dial tcp 10.134.0.138:8443: connect: connection refused"
Oct 29 23:00:29.083791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.083182    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg" podUID="e9fb824d-0d5d-4417-97ef-8aced007417a" containerName="payload-processing" containerID="cri-o://5ac499dd5f5eff583184484fc19fd27ee27af4a4a276f9af30d7c2ba6a6b9ed8" gracePeriod=25
Oct 29 23:00:29.291527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.291506    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 23:00:29.465961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.465541    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-kube-api-access-5p9rb\" (UniqueName: \"kubernetes.io/projected/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-kube-api-access-5p9rb\") pod \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") "
Oct 29 23:00:29.465961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.465623    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-openshift-service-ca\") pod \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") "
Oct 29 23:00:29.469041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.466606    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-metrics-certs\") pod \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") "
Oct 29 23:00:29.469041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.466859    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-maas-api-tls\") pod \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\" (UID: \"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf\") "
Oct 29 23:00:29.470253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.469952    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-openshift-service-ca" pod "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" (UID: "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:29.472615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.472567    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n" podUID="103c5747-89fb-46ce-9dde-c8fe9911e997" containerName="payload-pre-processing" containerID="cri-o://87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d" gracePeriod=25
Oct 29 23:00:29.479920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.479869    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-kube-api-access-5p9rb" pod "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" (UID: "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf"). InnerVolumeSpecName "kube-api-access-5p9rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:29.503447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.502738    4827 generic.go:356] "Generic (PLEG): container finished" podID="d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" containerID="7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135" exitCode=0
Oct 29 23:00:29.503447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.503103    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6" event={"ID":"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf","Type":"ContainerDied","Data":"7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135"}
Oct 29 23:00:29.503447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.503154    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6" event={"ID":"d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf","Type":"ContainerDied","Data":"5739a351ef0755120214a28ba48747d2b89c38dda49616b634d9a81616a9b432"}
Oct 29 23:00:29.503447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.503181    4827 scope.go:122] "RemoveContainer" containerID="7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135"
Oct 29 23:00:29.503447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.503310    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"
Oct 29 23:00:29.516055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.507252    4827 generic.go:356] "Generic (PLEG): container finished" podID="e9fb824d-0d5d-4417-97ef-8aced007417a" containerID="5ac499dd5f5eff583184484fc19fd27ee27af4a4a276f9af30d7c2ba6a6b9ed8" exitCode=0
Oct 29 23:00:29.516055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.507355    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg" event={"ID":"e9fb824d-0d5d-4417-97ef-8aced007417a","Type":"ContainerDied","Data":"5ac499dd5f5eff583184484fc19fd27ee27af4a4a276f9af30d7c2ba6a6b9ed8"}
Oct 29 23:00:29.574517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.573908    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-maas-api-tls" pod "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" (UID: "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:29.574517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.573999    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-metrics-certs" pod "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" (UID: "d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:29.574517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.574485    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.574517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.574509    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p9rb\" (UniqueName: \"kubernetes.io/projected/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-kube-api-access-5p9rb\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.574882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.574523    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.574882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.574537    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.580590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.580080    4827 scope.go:122] "RemoveContainer" containerID="7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135"
Oct 29 23:00:29.580590 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:29.580489    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135\": container with ID starting with 7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135 not found: ID does not exist" containerID="7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135"
Oct 29 23:00:29.580590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.580522    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135"} err="failed to get container status \"7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135\": rpc error: code = NotFound desc = could not find container \"7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135\": container with ID starting with 7a7e83ef94364a114715f4c9c37702683949820d47f377c61ba06dd2f5543135 not found: ID does not exist"
Oct 29 23:00:29.627060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.627032    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 23:00:29.777135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.777099    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fb824d-0d5d-4417-97ef-8aced007417a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fb824d-0d5d-4417-97ef-8aced007417a-plugins-config-volume\") pod \"e9fb824d-0d5d-4417-97ef-8aced007417a\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") "
Oct 29 23:00:29.777448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.777423    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fb824d-0d5d-4417-97ef-8aced007417a-kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/e9fb824d-0d5d-4417-97ef-8aced007417a-kube-api-access-zgqgf\") pod \"e9fb824d-0d5d-4417-97ef-8aced007417a\" (UID: \"e9fb824d-0d5d-4417-97ef-8aced007417a\") "
Oct 29 23:00:29.777556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.777518    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fb824d-0d5d-4417-97ef-8aced007417a-plugins-config-volume" pod "e9fb824d-0d5d-4417-97ef-8aced007417a" (UID: "e9fb824d-0d5d-4417-97ef-8aced007417a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:29.783389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.783359    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fb824d-0d5d-4417-97ef-8aced007417a-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.785153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.785123    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fb824d-0d5d-4417-97ef-8aced007417a-kube-api-access-zgqgf" pod "e9fb824d-0d5d-4417-97ef-8aced007417a" (UID: "e9fb824d-0d5d-4417-97ef-8aced007417a"). InnerVolumeSpecName "kube-api-access-zgqgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:29.818142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.818118    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 23:00:29.856432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.856401    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"]
Oct 29 23:00:29.864037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.864008    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-tjsb6"]
Oct 29 23:00:29.884937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.884903    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/e9fb824d-0d5d-4417-97ef-8aced007417a-kube-api-access-zgqgf\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.986387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.986315    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103c5747-89fb-46ce-9dde-c8fe9911e997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c5747-89fb-46ce-9dde-c8fe9911e997-plugins-config-volume\") pod \"103c5747-89fb-46ce-9dde-c8fe9911e997\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") "
Oct 29 23:00:29.986536 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.986514    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103c5747-89fb-46ce-9dde-c8fe9911e997-kube-api-access-hnxsv\" (UniqueName: \"kubernetes.io/projected/103c5747-89fb-46ce-9dde-c8fe9911e997-kube-api-access-hnxsv\") pod \"103c5747-89fb-46ce-9dde-c8fe9911e997\" (UID: \"103c5747-89fb-46ce-9dde-c8fe9911e997\") "
Oct 29 23:00:29.986686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.986659    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103c5747-89fb-46ce-9dde-c8fe9911e997-plugins-config-volume" pod "103c5747-89fb-46ce-9dde-c8fe9911e997" (UID: "103c5747-89fb-46ce-9dde-c8fe9911e997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:29.987137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.987115    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c5747-89fb-46ce-9dde-c8fe9911e997-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.988615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:29.988593    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103c5747-89fb-46ce-9dde-c8fe9911e997-kube-api-access-hnxsv" pod "103c5747-89fb-46ce-9dde-c8fe9911e997" (UID: "103c5747-89fb-46ce-9dde-c8fe9911e997"). InnerVolumeSpecName "kube-api-access-hnxsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:30.088954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.088924    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnxsv\" (UniqueName: \"kubernetes.io/projected/103c5747-89fb-46ce-9dde-c8fe9911e997-kube-api-access-hnxsv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:30.489972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.489942    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" path="/var/lib/kubelet/pods/d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf/volumes"
Oct 29 23:00:30.490495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.490481    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d" path="/var/lib/kubelet/pods/ec7274b6-9bbb-4cea-95c4-c6d8e7a7b75d/volumes"
Oct 29 23:00:30.517421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.517389    4827 generic.go:356] "Generic (PLEG): container finished" podID="103c5747-89fb-46ce-9dde-c8fe9911e997" containerID="87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d" exitCode=0
Oct 29 23:00:30.517567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.517455    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"
Oct 29 23:00:30.517567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.517453    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n" event={"ID":"103c5747-89fb-46ce-9dde-c8fe9911e997","Type":"ContainerDied","Data":"87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d"}
Oct 29 23:00:30.517667 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.517588    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n" event={"ID":"103c5747-89fb-46ce-9dde-c8fe9911e997","Type":"ContainerDied","Data":"8eec9a61ce92062e3d63f70832656668b0e266cb5ee17de1f989b1d84f4584ae"}
Oct 29 23:00:30.517667 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.517623    4827 scope.go:122] "RemoveContainer" containerID="87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d"
Oct 29 23:00:30.519381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.519270    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"
Oct 29 23:00:30.519381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.519265    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg" event={"ID":"e9fb824d-0d5d-4417-97ef-8aced007417a","Type":"ContainerDied","Data":"327d0a57bef21dc7b252d9d3390d60ef9494be292a0383261e3ec76d93ad04d0"}
Oct 29 23:00:30.530153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.530140    4827 scope.go:122] "RemoveContainer" containerID="87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d"
Oct 29 23:00:30.530454 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:30.530435    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d\": container with ID starting with 87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d not found: ID does not exist" containerID="87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d"
Oct 29 23:00:30.530518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.530460    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d"} err="failed to get container status \"87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d\": rpc error: code = NotFound desc = could not find container \"87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d\": container with ID starting with 87e377180ad2f21834236ff1eb1f52e987faebf5c24dbc1005d8857ffac8cf0d not found: ID does not exist"
Oct 29 23:00:30.530518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.530476    4827 scope.go:122] "RemoveContainer" containerID="5ac499dd5f5eff583184484fc19fd27ee27af4a4a276f9af30d7c2ba6a6b9ed8"
Oct 29 23:00:30.572060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.572031    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"]
Oct 29 23:00:30.577565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.577542    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0df9b0-5cfcd88cf5-hg47n"]
Oct 29 23:00:30.608520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.608493    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"]
Oct 29 23:00:30.614452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:30.614430    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0df9b0-769c7f9bdb-wnkwg"]
Oct 29 23:00:32.490240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:32.490213    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103c5747-89fb-46ce-9dde-c8fe9911e997" path="/var/lib/kubelet/pods/103c5747-89fb-46ce-9dde-c8fe9911e997/volumes"
Oct 29 23:00:32.490696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:32.490681    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fb824d-0d5d-4417-97ef-8aced007417a" path="/var/lib/kubelet/pods/e9fb824d-0d5d-4417-97ef-8aced007417a/volumes"
Oct 29 23:00:32.826606 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:32.826522    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:00:35.040477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:35.040436    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"]
Oct 29 23:00:35.050171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:35.050132    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19baf5dd-t55vz"]
Oct 29 23:00:36.054788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.054377    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"]
Oct 29 23:00:36.068785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.064346    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2912e-khj82"]
Oct 29 23:00:36.390989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.390958    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:36.495977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.495936    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef8870b-afab-4012-9e54-363671574c25" path="/var/lib/kubelet/pods/3ef8870b-afab-4012-9e54-363671574c25/volumes"
Oct 29 23:00:36.497013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.496987    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6c4480-bc04-476a-85f0-7b716349b07f" path="/var/lib/kubelet/pods/9a6c4480-bc04-476a-85f0-7b716349b07f/volumes"
Oct 29 23:00:36.554835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.554797    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"]
Oct 29 23:00:36.555747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555719    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:36.555747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555743    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555757    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8c28df07-1331-4dc8-90dc-43d93c8ddae1" containerName="payload-processing"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555776    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c28df07-1331-4dc8-90dc-43d93c8ddae1" containerName="payload-processing"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555805    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555812    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555822    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555828    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555835    4827 cpu_manager.go:414] "containerMap: removing container" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555840    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555859    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" containerName="maas-api"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555865    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" containerName="maas-api"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555875    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e9fb824d-0d5d-4417-97ef-8aced007417a" containerName="payload-processing"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555882    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fb824d-0d5d-4417-97ef-8aced007417a" containerName="payload-processing"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555905    4827 cpu_manager.go:414] "containerMap: removing container" podUID="28287833-a7b4-440c-b3e4-04c369bd615c" containerName="payload-pre-processing"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555910    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28287833-a7b4-440c-b3e4-04c369bd615c" containerName="payload-pre-processing"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555921    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555928    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fece158c-1f91-4810-9c88-df9aaf8ef8be" containerName="cleanup"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555939    4827 cpu_manager.go:414] "containerMap: removing container" podUID="103c5747-89fb-46ce-9dde-c8fe9911e997" containerName="payload-pre-processing"
Oct 29 23:00:36.555967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.555944    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103c5747-89fb-46ce-9dde-c8fe9911e997" containerName="payload-pre-processing"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556092    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe05cd7-9c9d-455a-8a8b-69c903af653e" containerName="cleanup"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556105    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="28287833-a7b4-440c-b3e4-04c369bd615c" containerName="payload-pre-processing"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556113    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="103c5747-89fb-46ce-9dde-c8fe9911e997" containerName="payload-pre-processing"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556121    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd33576d-8fab-40c9-8690-0f770df77547" containerName="cleanup"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556127    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c28df07-1331-4dc8-90dc-43d93c8ddae1" containerName="payload-processing"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556138    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ae80a-cc37-4511-9e1a-485554e20f43" containerName="cleanup"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556144    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerName="cleanup"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556150    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerName="cleanup"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556155    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="117c6362-84ed-4158-a670-9404fa625943" containerName="cleanup"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556162    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64b6dcf-a9a6-4ba5-9d2d-6907a0c0ebbf" containerName="maas-api"
Oct 29 23:00:36.556562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.556168    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fb824d-0d5d-4417-97ef-8aced007417a" containerName="payload-processing"
Oct 29 23:00:36.560298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.560275    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9e8482-aaa4-43c6-bf67-01087c5675b1-kube-api-access-zw6lq\" (UniqueName: \"kubernetes.io/projected/9c9e8482-aaa4-43c6-bf67-01087c5675b1-kube-api-access-zw6lq\") pod \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") "
Oct 29 23:00:36.560470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.560457    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9e8482-aaa4-43c6-bf67-01087c5675b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e8482-aaa4-43c6-bf67-01087c5675b1-maas-api-service-ca\") pod \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\" (UID: \"9c9e8482-aaa4-43c6-bf67-01087c5675b1\") "
Oct 29 23:00:36.560882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.560851    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9e8482-aaa4-43c6-bf67-01087c5675b1-maas-api-service-ca" pod "9c9e8482-aaa4-43c6-bf67-01087c5675b1" (UID: "9c9e8482-aaa4-43c6-bf67-01087c5675b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:36.561001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.560920    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.561362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.561341    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e8482-aaa4-43c6-bf67-01087c5675b1-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.562840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.562812    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9e8482-aaa4-43c6-bf67-01087c5675b1-kube-api-access-zw6lq" pod "9c9e8482-aaa4-43c6-bf67-01087c5675b1" (UID: "9c9e8482-aaa4-43c6-bf67-01087c5675b1"). InnerVolumeSpecName "kube-api-access-zw6lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:36.588792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.588749    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7d18f7-openshift-default-dockercfg-mbshf\""
Oct 29 23:00:36.595431 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.595375    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"]
Oct 29 23:00:36.662723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.662688    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6lq\" (UniqueName: \"kubernetes.io/projected/9c9e8482-aaa4-43c6-bf67-01087c5675b1-kube-api-access-zw6lq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.711805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.706665    4827 generic.go:356] "Generic (PLEG): container finished" podID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" containerID="401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289" exitCode=137
Oct 29 23:00:36.711805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.707123    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk" event={"ID":"9c9e8482-aaa4-43c6-bf67-01087c5675b1","Type":"ContainerDied","Data":"401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289"}
Oct 29 23:00:36.711805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.707188    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk" event={"ID":"9c9e8482-aaa4-43c6-bf67-01087c5675b1","Type":"ContainerDied","Data":"b9fdfe47910e3596a6d7b9793715cb85740ef15d8f0ce7761a9519e1e0626c22"}
Oct 29 23:00:36.711805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.707219    4827 scope.go:122] "RemoveContainer" containerID="401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289"
Oct 29 23:00:36.711805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.707412    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"
Oct 29 23:00:36.758158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.758130    4827 scope.go:122] "RemoveContainer" containerID="2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766211    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3d0b2-dc85-4627-a891-0c5d33639748-istiod-ca-cert\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766286    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-envoy\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766330    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-socket\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766396    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-token\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766528    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-certs\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766555    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-data\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766591    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-podinfo\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766717    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-credential-socket\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.775079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.766787    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4sh5\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-kube-api-access-k4sh5\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.885036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.880838    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900105    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-certs\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900156    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-data\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900199    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-podinfo\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900347    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-credential-socket\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900402    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4sh5\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-kube-api-access-k4sh5\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900576    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3d0b2-dc85-4627-a891-0c5d33639748-istiod-ca-cert\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900656    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-envoy\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900727    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-socket\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.900830    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-token\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.902306    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-certs\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.907793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.902587    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-data\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.920253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.911469    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-credential-socket\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.920253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.920146    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-socket\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.920253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.920202    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"]
Oct 29 23:00:36.933786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.924487    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3d0b2-dc85-4627-a891-0c5d33639748-istiod-ca-cert\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:36.958682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.958652    4827 scope.go:122] "RemoveContainer" containerID="401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289"
Oct 29 23:00:36.959500 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:36.959327    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289\": container with ID starting with 401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289 not found: ID does not exist" containerID="401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289"
Oct 29 23:00:36.959500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.959370    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289"} err="failed to get container status \"401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289\": rpc error: code = NotFound desc = could not find container \"401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289\": container with ID starting with 401b0ea516d64bffa9a714ae60f62883fa1f72d0bc1703a4e0f9939ecfe86289 not found: ID does not exist"
Oct 29 23:00:36.959500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.959395    4827 scope.go:122] "RemoveContainer" containerID="2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80"
Oct 29 23:00:36.960150 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:36.960064    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80\": container with ID starting with 2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80 not found: ID does not exist" containerID="2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80"
Oct 29 23:00:36.960150 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.960101    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80"} err="failed to get container status \"2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80\": rpc error: code = NotFound desc = could not find container \"2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80\": container with ID starting with 2fa81f9b78b12063a4b155e4541657b52ef202a98ecdf2c158600a6473367c80 not found: ID does not exist"
Oct 29 23:00:37.003789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:36.979888    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd525f-29888580-tlclk"]
Oct 29 23:00:37.027663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.027598    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-podinfo\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:37.028123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.028099    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-envoy\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:37.054793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.048852    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-token\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:37.114557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.114487    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4sh5\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-kube-api-access-k4sh5\") pod \"e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") " pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:37.299929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.212668    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:37.665612 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.665581    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"]
Oct 29 23:00:37.670826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.670804    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.685598 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.685570    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f173b1-serving-cert\""
Oct 29 23:00:37.697075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.697049    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"]
Oct 29 23:00:37.729166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.729133    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-metrics-certs\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.729343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.729279    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f12977a-6a23-47a5-be17-6af3c5b73688-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.729384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.729352    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6jr\" (UniqueName: \"kubernetes.io/projected/0f12977a-6a23-47a5-be17-6af3c5b73688-kube-api-access-lz6jr\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.729623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.729589    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.758021 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:37.757984    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd3d0b2_dc85_4627_a891_0c5d33639748.slice/crio-d685149585e990cbaf880c4b8ffff8efcaba5cdef76f179368008698cc07d9c0 WatchSource:0}: Error finding container d685149585e990cbaf880c4b8ffff8efcaba5cdef76f179368008698cc07d9c0: Status 404 returned error can't find the container with id d685149585e990cbaf880c4b8ffff8efcaba5cdef76f179368008698cc07d9c0
Oct 29 23:00:37.759276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.758857    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"]
Oct 29 23:00:37.761864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.761826    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:37.761950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.761904    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:37.761950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.761934    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:37.797193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.797167    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"]
Oct 29 23:00:37.802859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.802839    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.830616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.830587    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"]
Oct 29 23:00:37.831695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831518    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.831695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831583    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.831695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831691    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9dg\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-kube-api-access-zw9dg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.831988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831754    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-metrics-certs\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.831988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831833    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.831988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831900    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f12977a-6a23-47a5-be17-6af3c5b73688-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.831988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831954    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6jr\" (UniqueName: \"kubernetes.io/projected/0f12977a-6a23-47a5-be17-6af3c5b73688-kube-api-access-lz6jr\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.831988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.831986    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.851129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.832103    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30674ff2-2aeb-4768-9611-d61ea2e7eb80-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.851129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.832226    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.851129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.832274    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.851129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.834293    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f12977a-6a23-47a5-be17-6af3c5b73688-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.851129 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:37.834685    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-serving-cert: secret "maas-api-e2e-ipp-a-f173b1-serving-cert" not found
Oct 29 23:00:37.851129 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:37.834758    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls podName:0f12977a-6a23-47a5-be17-6af3c5b73688 nodeName:}" failed. No retries permitted until 2026-10-29 23:00:38.334739 +0000 UTC m=+1073.957650020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls") pod "maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j" (UID: "0f12977a-6a23-47a5-be17-6af3c5b73688") : secret "maas-api-e2e-ipp-a-f173b1-serving-cert" not found
Oct 29 23:00:37.851129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.835832    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-metrics-certs\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.891682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.891601    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6jr\" (UniqueName: \"kubernetes.io/projected/0f12977a-6a23-47a5-be17-6af3c5b73688-kube-api-access-lz6jr\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:37.956872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.954828    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.956872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.954899    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.956872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.955050    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9dg\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-kube-api-access-zw9dg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.956872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.955117    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.956872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.955284    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.956872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.955388    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30674ff2-2aeb-4768-9611-d61ea2e7eb80-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.956872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.955523    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:37.987866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:37.960059    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.056895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.005247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.056895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.021093    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9dg\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-kube-api-access-zw9dg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.056895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.023329    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.056895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.047420    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.066672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.066594    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.069434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.069408    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30674ff2-2aeb-4768-9611-d61ea2e7eb80-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vxstl\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.152811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.152688    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.365417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.365385    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:38.368160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.368134    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:38.481359 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:00:38.481322    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30674ff2_2aeb_4768_9611_d61ea2e7eb80.slice/crio-6b4f47bf081ded4537c08697fc321452579f3294dab5fbeb6777c1af531f3534 WatchSource:0}: Error finding container 6b4f47bf081ded4537c08697fc321452579f3294dab5fbeb6777c1af531f3534: Status 404 returned error can't find the container with id 6b4f47bf081ded4537c08697fc321452579f3294dab5fbeb6777c1af531f3534
Oct 29 23:00:38.484782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.484722    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:38.484897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.484825    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:38.493549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.493525    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9e8482-aaa4-43c6-bf67-01087c5675b1" path="/var/lib/kubelet/pods/9c9e8482-aaa4-43c6-bf67-01087c5675b1/volumes"
Oct 29 23:00:38.493955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.493923    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"]
Oct 29 23:00:38.610013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.605126    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:38.747363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.747330    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" event={"ID":"bbd3d0b2-dc85-4627-a891-0c5d33639748","Type":"ContainerStarted","Data":"56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118"}
Oct 29 23:00:38.747528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.747391    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" event={"ID":"bbd3d0b2-dc85-4627-a891-0c5d33639748","Type":"ContainerStarted","Data":"d685149585e990cbaf880c4b8ffff8efcaba5cdef76f179368008698cc07d9c0"}
Oct 29 23:00:38.750385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.750355    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl" event={"ID":"30674ff2-2aeb-4768-9611-d61ea2e7eb80","Type":"ContainerStarted","Data":"c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8"}
Oct 29 23:00:38.750514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.750404    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl" event={"ID":"30674ff2-2aeb-4768-9611-d61ea2e7eb80","Type":"ContainerStarted","Data":"6b4f47bf081ded4537c08697fc321452579f3294dab5fbeb6777c1af531f3534"}
Oct 29 23:00:38.751201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.751182    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:38.794862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.794812    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podStartSLOduration=2.794758998 podStartE2EDuration="2.794758998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:38.793187435 +0000 UTC m=+1074.416098455" watchObservedRunningTime="2026-10-29 23:00:38.794758998 +0000 UTC m=+1074.417670340"
Oct 29 23:00:38.847491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:38.847429    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl" podStartSLOduration=1.847412327 podStartE2EDuration="1.847412327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:38.839691329 +0000 UTC m=+1074.462602356" watchObservedRunningTime="2026-10-29 23:00:38.847412327 +0000 UTC m=+1074.470323350"
Oct 29 23:00:39.106974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.099797    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"]
Oct 29 23:00:39.291921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.291841    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:39.340834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.336924    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:39.772107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.772071    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j" event={"ID":"0f12977a-6a23-47a5-be17-6af3c5b73688","Type":"ContainerStarted","Data":"9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921"}
Oct 29 23:00:39.772304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.772167    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:39.772304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.772196    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j" event={"ID":"0f12977a-6a23-47a5-be17-6af3c5b73688","Type":"ContainerStarted","Data":"4d30d7ccb52ee09bd09d7fffd829aa28aa755e9d49b18c66682f9c91b784e7cb"}
Oct 29 23:00:39.772979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.772954    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:39.774722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.774481    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:00:39.774874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.774732    4827 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vxstl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:00:39.774874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:39.774787    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl" podUID="30674ff2-2aeb-4768-9611-d61ea2e7eb80" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:00:41.376111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:41.376062    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:41.433496 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:41.386617    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:00:41.438786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:41.388469    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j" podStartSLOduration=3.917784085 podStartE2EDuration="4.388450316s" totalImagesPullingTime="470.666231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:39.878150042 +0000 UTC m=+1075.501061062" watchObservedRunningTime="2026-10-29 23:00:41.388450316 +0000 UTC m=+1077.011361337"
Oct 29 23:00:41.628093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:41.598153    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"]
Oct 29 23:00:41.667429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:41.661753    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015a89d-tslcb"]
Oct 29 23:00:42.574793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:42.573206    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d76f3e7-bf83-4864-8331-994346725912" path="/var/lib/kubelet/pods/7d76f3e7-bf83-4864-8331-994346725912/volumes"
Oct 29 23:00:45.712794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:45.710124    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:45.822802 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:45.817106    4827 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.314s"
Oct 29 23:00:45.845683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:45.845649    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"]
Oct 29 23:00:46.060874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:46.052330    4827 scope.go:122] "RemoveContainer" containerID="acb7b7ebe2e75119aa4292ee606b27efd03b9dd108d100b1f8f045940c5f548b"
Oct 29 23:00:46.363883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:46.339894    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:00:46.375959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:46.375818    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"]
Oct 29 23:00:48.030127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:48.030094    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"]
Oct 29 23:00:48.041323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:48.041289    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c1025d-47gsq"]
Oct 29 23:00:48.490406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:48.490375    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9849bcba-1060-4fa9-8888-e6611efed472" path="/var/lib/kubelet/pods/9849bcba-1060-4fa9-8888-e6611efed472/volumes"
Oct 29 23:00:51.055694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.055656    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j" podUID="6823014e-1288-465d-b69e-0d74fc8a6bc0" containerName="payload-processing" containerID="cri-o://c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867" gracePeriod=25
Oct 29 23:00:51.391822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.386202    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p" podUID="f34adf7f-7dff-4502-ba91-099f0bf0be62" containerName="payload-pre-processing" containerID="cri-o://dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb" gracePeriod=25
Oct 29 23:00:51.508413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.508381    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 23:00:51.509734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.509577    4827 generic.go:356] "Generic (PLEG): container finished" podID="6823014e-1288-465d-b69e-0d74fc8a6bc0" containerID="c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867" exitCode=0
Oct 29 23:00:51.509921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.509893    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j" event={"ID":"6823014e-1288-465d-b69e-0d74fc8a6bc0","Type":"ContainerDied","Data":"c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867"}
Oct 29 23:00:51.509998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.509943    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j" event={"ID":"6823014e-1288-465d-b69e-0d74fc8a6bc0","Type":"ContainerDied","Data":"73020aad13d5ceacaf2ffd1f7ea63e4610da7eb5fc1859c258924776a7091de1"}
Oct 29 23:00:51.509998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.509966    4827 scope.go:122] "RemoveContainer" containerID="c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867"
Oct 29 23:00:51.520757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.520734    4827 scope.go:122] "RemoveContainer" containerID="c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867"
Oct 29 23:00:51.521109 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:51.521083    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867\": container with ID starting with c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867 not found: ID does not exist" containerID="c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867"
Oct 29 23:00:51.521109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.521117    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867"} err="failed to get container status \"c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867\": rpc error: code = NotFound desc = could not find container \"c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867\": container with ID starting with c60714733aa9814b9d5914cc6e55224d0e7e8ce356ed2703fbee1828a57d9867 not found: ID does not exist"
Oct 29 23:00:51.603830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.603352    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6823014e-1288-465d-b69e-0d74fc8a6bc0-kube-api-access-6xfm2\" (UniqueName: \"kubernetes.io/projected/6823014e-1288-465d-b69e-0d74fc8a6bc0-kube-api-access-6xfm2\") pod \"6823014e-1288-465d-b69e-0d74fc8a6bc0\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") "
Oct 29 23:00:51.603830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.603639    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6823014e-1288-465d-b69e-0d74fc8a6bc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823014e-1288-465d-b69e-0d74fc8a6bc0-plugins-config-volume\") pod \"6823014e-1288-465d-b69e-0d74fc8a6bc0\" (UID: \"6823014e-1288-465d-b69e-0d74fc8a6bc0\") "
Oct 29 23:00:51.605652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.605594    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6823014e-1288-465d-b69e-0d74fc8a6bc0-plugins-config-volume" pod "6823014e-1288-465d-b69e-0d74fc8a6bc0" (UID: "6823014e-1288-465d-b69e-0d74fc8a6bc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:51.607187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.607160    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6823014e-1288-465d-b69e-0d74fc8a6bc0-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:51.615359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.615328    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6823014e-1288-465d-b69e-0d74fc8a6bc0-kube-api-access-6xfm2" pod "6823014e-1288-465d-b69e-0d74fc8a6bc0" (UID: "6823014e-1288-465d-b69e-0d74fc8a6bc0"). InnerVolumeSpecName "kube-api-access-6xfm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:51.710570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.710482    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfm2\" (UniqueName: \"kubernetes.io/projected/6823014e-1288-465d-b69e-0d74fc8a6bc0-kube-api-access-6xfm2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:51.897464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.846983    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 23:00:51.989843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.981433    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34adf7f-7dff-4502-ba91-099f0bf0be62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34adf7f-7dff-4502-ba91-099f0bf0be62-plugins-config-volume\") pod \"f34adf7f-7dff-4502-ba91-099f0bf0be62\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") "
Oct 29 23:00:51.989843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:51.981784    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34adf7f-7dff-4502-ba91-099f0bf0be62-kube-api-access-ht2sn\" (UniqueName: \"kubernetes.io/projected/f34adf7f-7dff-4502-ba91-099f0bf0be62-kube-api-access-ht2sn\") pod \"f34adf7f-7dff-4502-ba91-099f0bf0be62\" (UID: \"f34adf7f-7dff-4502-ba91-099f0bf0be62\") "
Oct 29 23:00:52.040046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.015901    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34adf7f-7dff-4502-ba91-099f0bf0be62-plugins-config-volume" pod "f34adf7f-7dff-4502-ba91-099f0bf0be62" (UID: "f34adf7f-7dff-4502-ba91-099f0bf0be62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:52.150831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.150799    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34adf7f-7dff-4502-ba91-099f0bf0be62-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:52.167841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.151068    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"]
Oct 29 23:00:52.168104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.162267    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34adf7f-7dff-4502-ba91-099f0bf0be62-kube-api-access-ht2sn" pod "f34adf7f-7dff-4502-ba91-099f0bf0be62" (UID: "f34adf7f-7dff-4502-ba91-099f0bf0be62"). InnerVolumeSpecName "kube-api-access-ht2sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:52.243400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.243308    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2baec1-rpb2m"]
Oct 29 23:00:52.277302 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.277249    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2sn\" (UniqueName: \"kubernetes.io/projected/f34adf7f-7dff-4502-ba91-099f0bf0be62-kube-api-access-ht2sn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:52.502756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.502669    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd" path="/var/lib/kubelet/pods/95a7ac0f-a806-4bd8-bfe8-0c11f0f5dbfd/volumes"
Oct 29 23:00:52.524457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.523395    4827 generic.go:356] "Generic (PLEG): container finished" podID="f34adf7f-7dff-4502-ba91-099f0bf0be62" containerID="dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb" exitCode=0
Oct 29 23:00:52.524457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.523507    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p" event={"ID":"f34adf7f-7dff-4502-ba91-099f0bf0be62","Type":"ContainerDied","Data":"dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb"}
Oct 29 23:00:52.524457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.523554    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p" event={"ID":"f34adf7f-7dff-4502-ba91-099f0bf0be62","Type":"ContainerDied","Data":"9153b62e28d5bfce38fcaf07b8afda6801c72fafe519b00a2afd17cfdbf94c45"}
Oct 29 23:00:52.524457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.523576    4827 scope.go:122] "RemoveContainer" containerID="dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb"
Oct 29 23:00:52.524457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.523696    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"
Oct 29 23:00:52.528894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.528573    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"
Oct 29 23:00:52.609026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.608969    4827 scope.go:122] "RemoveContainer" containerID="dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb"
Oct 29 23:00:52.609562 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:52.609520    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb\": container with ID starting with dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb not found: ID does not exist" containerID="dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb"
Oct 29 23:00:52.609692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.609673    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb"} err="failed to get container status \"dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb\": rpc error: code = NotFound desc = could not find container \"dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb\": container with ID starting with dde51d31068788286f95b47547d3825ba7c86302f93676fb5bc1797e263fb4bb not found: ID does not exist"
Oct 29 23:00:52.615985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.615959    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"]
Oct 29 23:00:52.616256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.616208    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy" containerID="cri-o://85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f" gracePeriod=30
Oct 29 23:00:52.621060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.621003    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"]
Oct 29 23:00:52.621329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.621287    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy" containerID="cri-o://5c900aef2c40123ee0eadddddd5868e597af802f399de37304babc94b7db3f78" gracePeriod=30
Oct 29 23:00:52.633776 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.633741    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"]
Oct 29 23:00:52.635711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.635674    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy" containerID="cri-o://2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162" gracePeriod=30
Oct 29 23:00:52.642011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.641588    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"]
Oct 29 23:00:52.642011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.641845    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy" containerID="cri-o://fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb" gracePeriod=30
Oct 29 23:00:52.652351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.652325    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"]
Oct 29 23:00:52.660110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.660068    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-8fc9ffc5-6v48p"]
Oct 29 23:00:52.699794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.698278    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"]
Oct 29 23:00:52.711039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:52.711008    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-65579465c4-zzh5j"]
Oct 29 23:00:53.325816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:53.324846    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:00:53.325816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:53.324899    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:00:54.593846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:54.593410    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6823014e-1288-465d-b69e-0d74fc8a6bc0" path="/var/lib/kubelet/pods/6823014e-1288-465d-b69e-0d74fc8a6bc0/volumes"
Oct 29 23:00:54.594907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:54.594882    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34adf7f-7dff-4502-ba91-099f0bf0be62" path="/var/lib/kubelet/pods/f34adf7f-7dff-4502-ba91-099f0bf0be62/volumes"
Oct 29 23:00:55.720669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:55.720633    4827 generic.go:356] "Generic (PLEG): container finished" podID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerID="978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb" exitCode=0
Oct 29 23:00:55.721181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:55.720706    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" event={"ID":"32fadd7a-083a-4b36-8d31-6d89fd4e9739","Type":"ContainerDied","Data":"978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb"}
Oct 29 23:00:55.721181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:55.721177    4827 scope.go:122] "RemoveContainer" containerID="978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb"
Oct 29 23:00:55.723171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:55.723142    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:55.723261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:55.723212    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:55.723261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:55.723243    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:00:56.140972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:56.140942    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 23:00:56.734888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:56.734856    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" event={"ID":"32fadd7a-083a-4b36-8d31-6d89fd4e9739","Type":"ContainerStarted","Data":"385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad"}
Oct 29 23:00:57.314070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:57.314031    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 23:00:57.315073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:57.315049    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 29 23:00:57.315286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:57.315115    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 29 23:00:57.814135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:57.813532    4827 generic.go:356] "Generic (PLEG): container finished" podID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerID="5c900aef2c40123ee0eadddddd5868e597af802f399de37304babc94b7db3f78" exitCode=0
Oct 29 23:00:57.814135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:57.813905    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" event={"ID":"05d3629c-1694-4cb8-86ad-46803f41dfc0","Type":"ContainerDied","Data":"5c900aef2c40123ee0eadddddd5868e597af802f399de37304babc94b7db3f78"}
Oct 29 23:00:57.814135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:57.814010    4827 scope.go:122] "RemoveContainer" containerID="9f1c4b6a33310223e33da26d618a1278935d44c8c5a175ab0c72ac269c333b9e"
Oct 29 23:00:58.240399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.238120    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 23:00:58.420157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.416256    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 29 23:00:58.420157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.416318    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 29 23:00:58.452901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.452372    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-token\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-token\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.452901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.452544    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-data\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.452901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.452594    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-podinfo\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.452901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.452709    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-kube-api-access-vmtg9\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-kube-api-access-vmtg9\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.452901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.452793    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-envoy\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.453281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.452941    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istiod-ca-cert\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.453281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.453008    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-credential-socket\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.453281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.453060    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-certs\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.453281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.453101    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-socket\") pod \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\" (UID: \"4a80dafa-e4e6-43b0-979e-e91171c0ea24\") "
Oct 29 23:00:58.478788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.458517    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-socket" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.497838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.495084    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istiod-ca-cert" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:58.520602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.520557    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-credential-socket" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.528091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.521435    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-certs" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.528091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.521882    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-data" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.544085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.542894    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-podinfo" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:58.553364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.553311    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-kube-api-access-vmtg9" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "kube-api-access-vmtg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.553834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.553793    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-token" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.555004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.554975    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-envoy" pod "4a80dafa-e4e6-43b0-979e-e91171c0ea24" (UID: "4a80dafa-e4e6-43b0-979e-e91171c0ea24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555495    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555523    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555536    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtg9\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-kube-api-access-vmtg9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555559    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555572    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555584    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555597    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555609    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a80dafa-e4e6-43b0-979e-e91171c0ea24-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.555947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.555621    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a80dafa-e4e6-43b0-979e-e91171c0ea24-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.586977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.586947    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 23:00:58.598719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.598693    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659444    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-credential-socket\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659496    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-socket\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659560    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-podinfo\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659617    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-token\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-token\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659672    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-data\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659735    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74e36796-fc78-472c-89da-3eb09337333f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e36796-fc78-472c-89da-3eb09337333f-istio-podinfo\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659803    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-envoy\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659902    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-istio-token\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-istio-token\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.659977    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f7a705-d171-4bd8-a071-4f4ccd575485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f7a705-d171-4bd8-a071-4f4ccd575485-istiod-ca-cert\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660076    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-certs\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660087    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-socket" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660137    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-envoy\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660098    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-credential-socket" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660173    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-data" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660201    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-credential-socket\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660271    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-socket\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660412    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-kube-api-access-n7qw2\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-kube-api-access-n7qw2\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660449    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-certs\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660842    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-certs" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.660993    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f7a705-d171-4bd8-a071-4f4ccd575485-istiod-ca-cert" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.661259    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-socket" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.661457    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-credential-socket" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.662444    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-kube-api-access-qjjjv\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-kube-api-access-qjjjv\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.662500    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-data\") pod \"a0f7a705-d171-4bd8-a071-4f4ccd575485\" (UID: \"a0f7a705-d171-4bd8-a071-4f4ccd575485\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.662681    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e36796-fc78-472c-89da-3eb09337333f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e36796-fc78-472c-89da-3eb09337333f-istiod-ca-cert\") pod \"74e36796-fc78-472c-89da-3eb09337333f\" (UID: \"74e36796-fc78-472c-89da-3eb09337333f\") "
Oct 29 23:00:58.671292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.669384    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-envoy" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.673528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.673297    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-kube-api-access-qjjjv" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "kube-api-access-qjjjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.673528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.673495    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-data" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.673995    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e36796-fc78-472c-89da-3eb09337333f-istiod-ca-cert" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.689939    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-certs" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.696245    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74e36796-fc78-472c-89da-3eb09337333f-istio-podinfo" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.696335    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-envoy" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698862    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f7a705-d171-4bd8-a071-4f4ccd575485-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698888    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698905    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698920    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698934    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698947    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698962    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.703736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.698975    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e36796-fc78-472c-89da-3eb09337333f-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.711309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.711278    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-istio-token" pod "74e36796-fc78-472c-89da-3eb09337333f" (UID: "74e36796-fc78-472c-89da-3eb09337333f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.711420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.711388    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-kube-api-access-n7qw2" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "kube-api-access-n7qw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.712052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.712030    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-podinfo" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:58.712645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.712618    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-token" pod "a0f7a705-d171-4bd8-a071-4f4ccd575485" (UID: "a0f7a705-d171-4bd8-a071-4f4ccd575485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.752383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.752107    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.803730    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e36796-fc78-472c-89da-3eb09337333f-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807016    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807061    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807078    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e36796-fc78-472c-89da-3eb09337333f-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807091    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807102    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807113    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807125    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qw2\" (UniqueName: \"kubernetes.io/projected/a0f7a705-d171-4bd8-a071-4f4ccd575485-kube-api-access-n7qw2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807139    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjjv\" (UniqueName: \"kubernetes.io/projected/74e36796-fc78-472c-89da-3eb09337333f-kube-api-access-qjjjv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.807180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.807151    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f7a705-d171-4bd8-a071-4f4ccd575485-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.837864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.836006    4827 generic.go:356] "Generic (PLEG): container finished" podID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerID="2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162" exitCode=0
Oct 29 23:00:58.837864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.836095    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" event={"ID":"a0f7a705-d171-4bd8-a071-4f4ccd575485","Type":"ContainerDied","Data":"2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162"}
Oct 29 23:00:58.837864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.836146    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4" event={"ID":"a0f7a705-d171-4bd8-a071-4f4ccd575485","Type":"ContainerDied","Data":"8d000efd3100fc59c88b58d8692b053fe96252805562fd7d538b40beb2d52bc8"}
Oct 29 23:00:58.837864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.836176    4827 scope.go:122] "RemoveContainer" containerID="2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162"
Oct 29 23:00:58.837864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.836299    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"
Oct 29 23:00:58.852155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.840008    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2" event={"ID":"05d3629c-1694-4cb8-86ad-46803f41dfc0","Type":"ContainerDied","Data":"f3e9cb12af550c2f1dc9c8d0bd648f67a415bf932021c624219f11c402bbed0e"}
Oct 29 23:00:58.852155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.840159    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"
Oct 29 23:00:58.852155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.844149    4827 generic.go:356] "Generic (PLEG): container finished" podID="74e36796-fc78-472c-89da-3eb09337333f" containerID="85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f" exitCode=0
Oct 29 23:00:58.852155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.844266    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" event={"ID":"74e36796-fc78-472c-89da-3eb09337333f","Type":"ContainerDied","Data":"85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f"}
Oct 29 23:00:58.852155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.844313    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747" event={"ID":"74e36796-fc78-472c-89da-3eb09337333f","Type":"ContainerDied","Data":"ef592e3f8e363c2f32ee57d47dd025a1a4bdb759e8b005990ab626e28e7b535a"}
Oct 29 23:00:58.852155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.844392    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"
Oct 29 23:00:58.890243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.862016    4827 generic.go:356] "Generic (PLEG): container finished" podID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerID="fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb" exitCode=0
Oct 29 23:00:58.890243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.863306    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"
Oct 29 23:00:58.890243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.875979    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" event={"ID":"4a80dafa-e4e6-43b0-979e-e91171c0ea24","Type":"ContainerDied","Data":"fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb"}
Oct 29 23:00:58.890243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.876038    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj" event={"ID":"4a80dafa-e4e6-43b0-979e-e91171c0ea24","Type":"ContainerDied","Data":"fa5f6f31b51555ee5a13c9cec8749bb71ec85f1e406a82deadae3877a3ace21a"}
Oct 29 23:00:58.891957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.891937    4827 scope.go:122] "RemoveContainer" containerID="2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162"
Oct 29 23:00:58.893576 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:58.893474    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162\": container with ID starting with 2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162 not found: ID does not exist" containerID="2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162"
Oct 29 23:00:58.893576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.893502    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162"} err="failed to get container status \"2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162\": rpc error: code = NotFound desc = could not find container \"2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162\": container with ID starting with 2454a2276c8cadf272227d622ddbff2604e66dbdc6c5befa3103d77ebd17d162 not found: ID does not exist"
Oct 29 23:00:58.893576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.893518    4827 scope.go:122] "RemoveContainer" containerID="5c900aef2c40123ee0eadddddd5868e597af802f399de37304babc94b7db3f78"
Oct 29 23:00:58.907520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.907499    4827 scope.go:122] "RemoveContainer" containerID="85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f"
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.908424    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d3629c-1694-4cb8-86ad-46803f41dfc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d3629c-1694-4cb8-86ad-46803f41dfc0-istiod-ca-cert\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.908567    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-podinfo\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.908649    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-socket\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.908819    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-kube-api-access-5p7kt\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-kube-api-access-5p7kt\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.908886    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-token\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-token\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.908947    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-certs\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.909024    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d3629c-1694-4cb8-86ad-46803f41dfc0-istiod-ca-cert" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.909169    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-data\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.909433    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-data" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.909196    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-envoy\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.909531    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-socket" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.909793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.909593    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-certs" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.910811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.909890    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-credential-socket\") pod \"05d3629c-1694-4cb8-86ad-46803f41dfc0\" (UID: \"05d3629c-1694-4cb8-86ad-46803f41dfc0\") "
Oct 29 23:00:58.910811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.910155    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-credential-socket" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.911560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.910960    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.911560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.910982    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.911560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.910997    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d3629c-1694-4cb8-86ad-46803f41dfc0-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.911560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.911009    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.911560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.911020    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.912316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.911974    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-podinfo" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:58.912316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.912042    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-kube-api-access-5p7kt" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "kube-api-access-5p7kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.912848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.912787    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-token" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.913933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.913914    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-envoy" pod "05d3629c-1694-4cb8-86ad-46803f41dfc0" (UID: "05d3629c-1694-4cb8-86ad-46803f41dfc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:58.991385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.991271    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"]
Oct 29 23:00:58.993123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.993030    4827 scope.go:122] "RemoveContainer" containerID="85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f"
Oct 29 23:00:58.993918 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:58.993490    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f\": container with ID starting with 85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f not found: ID does not exist" containerID="85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f"
Oct 29 23:00:58.993918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.993524    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f"} err="failed to get container status \"85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f\": rpc error: code = NotFound desc = could not find container \"85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f\": container with ID starting with 85d8dd1f1c6c5b6af36d6683981ae0145a0940dc522d36db091681e2ed5ed01f not found: ID does not exist"
Oct 29 23:00:58.993918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.993583    4827 scope.go:122] "RemoveContainer" containerID="fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb"
Oct 29 23:00:58.999376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:58.999346    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-rzwh4"]
Oct 29 23:00:59.004677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.004663    4827 scope.go:122] "RemoveContainer" containerID="fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb"
Oct 29 23:00:59.004969 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:00:59.004952    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb\": container with ID starting with fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb not found: ID does not exist" containerID="fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb"
Oct 29 23:00:59.005023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.004977    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb"} err="failed to get container status \"fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb\": rpc error: code = NotFound desc = could not find container \"fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb\": container with ID starting with fc817491bf749b382ed7630ce1f8b384af18e1df53f09a9476a1f6e2d0f2b4bb not found: ID does not exist"
Oct 29 23:00:59.013585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.013557    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7kt\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-kube-api-access-5p7kt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.013679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.013587    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.013679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.013602    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.013679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.013653    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d3629c-1694-4cb8-86ad-46803f41dfc0-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.017161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.017139    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"]
Oct 29 23:00:59.022479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.022457    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-w2747"]
Oct 29 23:00:59.054917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.054893    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"]
Oct 29 23:00:59.058911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.058892    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-gf5rj"]
Oct 29 23:00:59.201927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.201898    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"]
Oct 29 23:00:59.206436 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.206412    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fk6d2"]
Oct 29 23:00:59.314935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.314858    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 29 23:00:59.314935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:00:59.314907    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 29 23:01:00.314228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.314190    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 29 23:01:00.314693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.314254    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 29 23:01:00.500474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.500442    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" path="/var/lib/kubelet/pods/05d3629c-1694-4cb8-86ad-46803f41dfc0/volumes"
Oct 29 23:01:00.501042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.501026    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" path="/var/lib/kubelet/pods/4a80dafa-e4e6-43b0-979e-e91171c0ea24/volumes"
Oct 29 23:01:00.501499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.501486    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e36796-fc78-472c-89da-3eb09337333f" path="/var/lib/kubelet/pods/74e36796-fc78-472c-89da-3eb09337333f/volumes"
Oct 29 23:01:00.502272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.502256    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" path="/var/lib/kubelet/pods/a0f7a705-d171-4bd8-a071-4f4ccd575485/volumes"
Oct 29 23:01:00.723921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.723889    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:01:00.724099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:00.724017    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:01:01.314618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.314582    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 29 23:01:01.315128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.314657    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 29 23:01:01.997302 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.997270    4827 generic.go:356] "Generic (PLEG): container finished" podID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerID="889125e7031170fddd986093a37a3dbac36ed904375dd0fe4805ae8545935eb1" exitCode=0
Oct 29 23:01:01.997476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.997413    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" event={"ID":"469b8d53-8386-4f8b-942b-d0fedb13eb95","Type":"ContainerDied","Data":"889125e7031170fddd986093a37a3dbac36ed904375dd0fe4805ae8545935eb1"}
Oct 29 23:01:01.997929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.997913    4827 scope.go:122] "RemoveContainer" containerID="889125e7031170fddd986093a37a3dbac36ed904375dd0fe4805ae8545935eb1"
Oct 29 23:01:01.999903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.999870    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:02.000013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.999947    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:02.000013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:01.999986    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:02.314129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:02.314032    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 29 23:01:02.314129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:02.314101    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 29 23:01:03.065511 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:03.065467    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" event={"ID":"469b8d53-8386-4f8b-942b-d0fedb13eb95","Type":"ContainerStarted","Data":"a85607ed531bd3606b59a547df30c08275c1197a533d2b082f08f93edf63631d"}
Oct 29 23:01:03.078077 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:01:03.078041    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:01:03.168122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:03.168089    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 23:01:03.168500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:03.168409    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 23:01:03.168668 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:03.168645    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 23:01:03.509619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:03.509510    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 23:01:03.509985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:03.509963    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 23:01:03.511650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:03.511627    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 23:01:04.168169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:04.168133    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 23:01:04.168953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:04.168198    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 23:01:05.182199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:05.182163    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 23:01:05.183179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:05.182791    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 23:01:06.186130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:06.186091    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 23:01:06.186738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:06.186705    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 23:01:07.180791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:07.178162    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 23:01:07.180791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:07.178234    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 23:01:08.180796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:08.177158    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 23:01:08.180796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:08.177220    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 23:01:09.169437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:09.169408    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 23:01:09.169907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:09.169884    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 23:01:09.170651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:09.170632    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 23:01:10.419903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.415658    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"]
Oct 29 23:01:10.452788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.435982    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy" containerID="cri-o://816ced82fd1b6237163942c91741aa892253a6e3cc4eb57c3f922ec8f576cc39" gracePeriod=30
Oct 29 23:01:10.752417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.752364    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"]
Oct 29 23:01:10.753738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753702    4827 cpu_manager.go:414] "containerMap: removing container" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy"
Oct 29 23:01:10.753898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753743    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy"
Oct 29 23:01:10.753898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753789    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f34adf7f-7dff-4502-ba91-099f0bf0be62" containerName="payload-pre-processing"
Oct 29 23:01:10.753898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753797    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34adf7f-7dff-4502-ba91-099f0bf0be62" containerName="payload-pre-processing"
Oct 29 23:01:10.753898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753811    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy"
Oct 29 23:01:10.753898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753819    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy"
Oct 29 23:01:10.753898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753861    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy"
Oct 29 23:01:10.753898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753872    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753901    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6823014e-1288-465d-b69e-0d74fc8a6bc0" containerName="payload-processing"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753912    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6823014e-1288-465d-b69e-0d74fc8a6bc0" containerName="payload-processing"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753945    4827 cpu_manager.go:414] "containerMap: removing container" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.753956    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.754150    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a80dafa-e4e6-43b0-979e-e91171c0ea24" containerName="istio-proxy"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.754164    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.754181    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f7a705-d171-4bd8-a071-4f4ccd575485" containerName="istio-proxy"
Oct 29 23:01:10.754217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.754201    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6823014e-1288-465d-b69e-0d74fc8a6bc0" containerName="payload-processing"
Oct 29 23:01:10.754475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.754223    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e36796-fc78-472c-89da-3eb09337333f" containerName="istio-proxy"
Oct 29 23:01:10.754475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.754247    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34adf7f-7dff-4502-ba91-099f0bf0be62" containerName="payload-pre-processing"
Oct 29 23:01:10.757702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.757675    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:10.758168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.758134    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"]
Oct 29 23:01:10.772581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.772563    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7d18f7-serving-cert\""
Oct 29 23:01:10.896470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.896425    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:10.896735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.896710    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50ad01a-0312-452c-997f-e928048a2c06-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:10.896910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.896811    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:10.896910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:10.896846    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvnd\" (UniqueName: \"kubernetes.io/projected/b50ad01a-0312-452c-997f-e928048a2c06-kube-api-access-sdvnd\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.002293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.000986    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.002293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.001269    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50ad01a-0312-452c-997f-e928048a2c06-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.002293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.001340    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.002293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.001376    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvnd\" (UniqueName: \"kubernetes.io/projected/b50ad01a-0312-452c-997f-e928048a2c06-kube-api-access-sdvnd\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.003011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.002987    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50ad01a-0312-452c-997f-e928048a2c06-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.006189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.005722    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.006189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.005906    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.024549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.024521    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvnd\" (UniqueName: \"kubernetes.io/projected/b50ad01a-0312-452c-997f-e928048a2c06-kube-api-access-sdvnd\") pod \"maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.074288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.074261    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:11.389028 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:01:11.388990    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50ad01a_0312_452c_997f_e928048a2c06.slice/crio-908eb58a3168178b2964ea3360d86e989e195daaaebf60d0b045b9614a3950df WatchSource:0}: Error finding container 908eb58a3168178b2964ea3360d86e989e195daaaebf60d0b045b9614a3950df: Status 404 returned error can't find the container with id 908eb58a3168178b2964ea3360d86e989e195daaaebf60d0b045b9614a3950df
Oct 29 23:01:11.389672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:11.389644    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"]
Oct 29 23:01:12.205484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:12.205446    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97" event={"ID":"b50ad01a-0312-452c-997f-e928048a2c06","Type":"ContainerStarted","Data":"4bb28726cdd9c1d5d3d459cd93882b8bacd343db174d84472615108f975d0232"}
Oct 29 23:01:12.205970 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:12.205519    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97" event={"ID":"b50ad01a-0312-452c-997f-e928048a2c06","Type":"ContainerStarted","Data":"908eb58a3168178b2964ea3360d86e989e195daaaebf60d0b045b9614a3950df"}
Oct 29 23:01:12.205970 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:12.205548    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:14.032844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:14.032757    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97" podStartSLOduration=3.67252649 podStartE2EDuration="4.032742902s" totalImagesPullingTime="360.216412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:12.290131712 +0000 UTC m=+1107.913042761" watchObservedRunningTime="2026-10-29 23:01:14.032742902 +0000 UTC m=+1109.655653923"
Oct 29 23:01:14.037485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:14.037458    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"]
Oct 29 23:01:14.052028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:14.051996    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0a3f00-cdcrw"]
Oct 29 23:01:14.741911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:14.739498    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570f1185-c7dd-4098-b05d-ce776a83926e" path="/var/lib/kubelet/pods/570f1185-c7dd-4098-b05d-ce776a83926e/volumes"
Oct 29 23:01:15.280936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.274393    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"]
Oct 29 23:01:15.280936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.274673    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm" podUID="02a2c601-a75e-439a-9400-9ee9e86c8c6a" containerName="maas-api" containerID="cri-o://a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67" gracePeriod=30
Oct 29 23:01:15.410095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.410034    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"]
Oct 29 23:01:15.446634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.446601    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"]
Oct 29 23:01:15.593975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.593946    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 23:01:15.666104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.666077    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-maas-api-tls\") pod \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") "
Oct 29 23:01:15.666351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.666336    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a2c601-a75e-439a-9400-9ee9e86c8c6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a2c601-a75e-439a-9400-9ee9e86c8c6a-openshift-service-ca\") pod \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") "
Oct 29 23:01:15.666421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.666412    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a2c601-a75e-439a-9400-9ee9e86c8c6a-kube-api-access-t5g97\" (UniqueName: \"kubernetes.io/projected/02a2c601-a75e-439a-9400-9ee9e86c8c6a-kube-api-access-t5g97\") pod \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") "
Oct 29 23:01:15.666488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.666472    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-metrics-certs\") pod \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\" (UID: \"02a2c601-a75e-439a-9400-9ee9e86c8c6a\") "
Oct 29 23:01:15.666804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.666736    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a2c601-a75e-439a-9400-9ee9e86c8c6a-openshift-service-ca" pod "02a2c601-a75e-439a-9400-9ee9e86c8c6a" (UID: "02a2c601-a75e-439a-9400-9ee9e86c8c6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:15.667340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.667287    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a2c601-a75e-439a-9400-9ee9e86c8c6a-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.668523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.668339    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-maas-api-tls" pod "02a2c601-a75e-439a-9400-9ee9e86c8c6a" (UID: "02a2c601-a75e-439a-9400-9ee9e86c8c6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:15.668956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.668925    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a2c601-a75e-439a-9400-9ee9e86c8c6a-kube-api-access-t5g97" pod "02a2c601-a75e-439a-9400-9ee9e86c8c6a" (UID: "02a2c601-a75e-439a-9400-9ee9e86c8c6a"). InnerVolumeSpecName "kube-api-access-t5g97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:15.669093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.668997    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-metrics-certs" pod "02a2c601-a75e-439a-9400-9ee9e86c8c6a" (UID: "02a2c601-a75e-439a-9400-9ee9e86c8c6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:15.769357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.769329    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5g97\" (UniqueName: \"kubernetes.io/projected/02a2c601-a75e-439a-9400-9ee9e86c8c6a-kube-api-access-t5g97\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.769357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.769358    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.769497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:15.769367    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a2c601-a75e-439a-9400-9ee9e86c8c6a-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:16.300921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.300794    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"]
Oct 29 23:01:16.525791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.524046    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0df9b0-p6z9q"]
Oct 29 23:01:16.847871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.847834    4827 generic.go:356] "Generic (PLEG): container finished" podID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerID="816ced82fd1b6237163942c91741aa892253a6e3cc4eb57c3f922ec8f576cc39" exitCode=0
Oct 29 23:01:16.852624 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.852596    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f146ab2e-c76c-46c7-8e0f-0379c50155da" path="/var/lib/kubelet/pods/f146ab2e-c76c-46c7-8e0f-0379c50155da/volumes"
Oct 29 23:01:16.858555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.858529    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 23:01:16.865799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.857275    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"
Oct 29 23:01:16.876813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.857208    4827 generic.go:356] "Generic (PLEG): container finished" podID="02a2c601-a75e-439a-9400-9ee9e86c8c6a" containerID="a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67" exitCode=0
Oct 29 23:01:16.888179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.888134    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" event={"ID":"5a5f0ed7-ce14-4995-be27-a38b35058656","Type":"ContainerDied","Data":"816ced82fd1b6237163942c91741aa892253a6e3cc4eb57c3f922ec8f576cc39"}
Oct 29 23:01:16.888363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.888206    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm" event={"ID":"02a2c601-a75e-439a-9400-9ee9e86c8c6a","Type":"ContainerDied","Data":"a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67"}
Oct 29 23:01:16.888363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.888238    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm" event={"ID":"02a2c601-a75e-439a-9400-9ee9e86c8c6a","Type":"ContainerDied","Data":"8f9d48e5876648f57c5017d24539d8191038e8ee292ea593507e1912eb6b17a4"}
Oct 29 23:01:16.888363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:16.888269    4827 scope.go:122] "RemoveContainer" containerID="816ced82fd1b6237163942c91741aa892253a6e3cc4eb57c3f922ec8f576cc39"
Oct 29 23:01:17.085380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.085346    4827 scope.go:122] "RemoveContainer" containerID="a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67"
Oct 29 23:01:17.195801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.195473    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-token\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-token\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.195801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.195551    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-podinfo\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.195801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.195596    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-credential-socket\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.198278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.196633    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-envoy\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.198278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.196719    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-certs\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.198278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.196808    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-socket\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.198278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.196952    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5f0ed7-ce14-4995-be27-a38b35058656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f0ed7-ce14-4995-be27-a38b35058656-istiod-ca-cert\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.198278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.197019    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-kube-api-access-7t7m8\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-kube-api-access-7t7m8\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.198278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.197054    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-data\") pod \"5a5f0ed7-ce14-4995-be27-a38b35058656\" (UID: \"5a5f0ed7-ce14-4995-be27-a38b35058656\") "
Oct 29 23:01:17.206666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.206615    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-data" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:17.219341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.219270    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-token" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:17.227158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.227122    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-certs" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:17.227567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.227521    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-credential-socket" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:17.233008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.231744    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5f0ed7-ce14-4995-be27-a38b35058656-istiod-ca-cert" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:17.234261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.234231    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-socket" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:17.240618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.236693    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-podinfo" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:17.251073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.250785    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-envoy" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:17.259032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.259005    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"]
Oct 29 23:01:17.262042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.262021    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328a78a3-jwr5b"]
Oct 29 23:01:17.263013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.262988    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-kube-api-access-7t7m8" pod "5a5f0ed7-ce14-4995-be27-a38b35058656" (UID: "5a5f0ed7-ce14-4995-be27-a38b35058656"). InnerVolumeSpecName "kube-api-access-7t7m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:17.265741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.265710    4827 scope.go:122] "RemoveContainer" containerID="a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67"
Oct 29 23:01:17.266069 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:01:17.266049    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67\": container with ID starting with a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67 not found: ID does not exist" containerID="a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67"
Oct 29 23:01:17.266126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.266076    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67"} err="failed to get container status \"a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67\": rpc error: code = NotFound desc = could not find container \"a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67\": container with ID starting with a6f2ae4fc28dd64788ed27ecb2147ef3252c500696fa9f3a154486bab6a21e67 not found: ID does not exist"
Oct 29 23:01:17.274284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.274262    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"]
Oct 29 23:01:17.278361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.278338    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0df9b0-c9fb6fcb5-d8jsm"]
Oct 29 23:01:17.307921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307901    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307924    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307933    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307947    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307959    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307971    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307982    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f0ed7-ce14-4995-be27-a38b35058656-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307989    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t7m8\" (UniqueName: \"kubernetes.io/projected/5a5f0ed7-ce14-4995-be27-a38b35058656-kube-api-access-7t7m8\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.308209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.307998    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5f0ed7-ce14-4995-be27-a38b35058656-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.886699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.886657    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4" event={"ID":"5a5f0ed7-ce14-4995-be27-a38b35058656","Type":"ContainerDied","Data":"0e7af7f70122dc42dfe80c46a95ea8704321938ba912da3ddc8b19e9b70cf1b6"}
Oct 29 23:01:17.886903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.886679    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"
Oct 29 23:01:17.940291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.940266    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"]
Oct 29 23:01:17.952629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:17.952606    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-68cx4"]
Oct 29 23:01:18.275992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:18.275920    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:01:18.504674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:18.504384    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a2c601-a75e-439a-9400-9ee9e86c8c6a" path="/var/lib/kubelet/pods/02a2c601-a75e-439a-9400-9ee9e86c8c6a/volumes"
Oct 29 23:01:18.507876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:18.505297    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" path="/var/lib/kubelet/pods/5a5f0ed7-ce14-4995-be27-a38b35058656/volumes"
Oct 29 23:01:18.507876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:18.506068    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c13d60d-4d6a-4ce0-ba8b-76eea17002e2" path="/var/lib/kubelet/pods/7c13d60d-4d6a-4ce0-ba8b-76eea17002e2/volumes"
Oct 29 23:01:19.622423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:19.622389    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"]
Oct 29 23:01:19.622971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:19.622618    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy" containerID="cri-o://e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0" gracePeriod=30
Oct 29 23:01:20.411051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.411011    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx" podUID="f0e3f789-5057-4149-adae-d626175767e0" containerName="payload-processing" containerID="cri-o://ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d" gracePeriod=25
Oct 29 23:01:20.447230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.447199    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl" podUID="bf0fa94f-5748-46f8-adeb-8ad243f3e72d" containerName="payload-pre-processing" containerID="cri-o://c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65" gracePeriod=25
Oct 29 23:01:20.804640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.804612    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:01:20.836214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.836194    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:01:20.931305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.931228    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e3f789-5057-4149-adae-d626175767e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e3f789-5057-4149-adae-d626175767e0-plugins-config-volume\") pod \"f0e3f789-5057-4149-adae-d626175767e0\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") "
Oct 29 23:01:20.931305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.931266    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-kube-api-access-jrqpg\" (UniqueName: \"kubernetes.io/projected/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-kube-api-access-jrqpg\") pod \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") "
Oct 29 23:01:20.931522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.931397    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-plugins-config-volume\") pod \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\" (UID: \"bf0fa94f-5748-46f8-adeb-8ad243f3e72d\") "
Oct 29 23:01:20.931522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.931497    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e3f789-5057-4149-adae-d626175767e0-kube-api-access-hr7jl\" (UniqueName: \"kubernetes.io/projected/f0e3f789-5057-4149-adae-d626175767e0-kube-api-access-hr7jl\") pod \"f0e3f789-5057-4149-adae-d626175767e0\" (UID: \"f0e3f789-5057-4149-adae-d626175767e0\") "
Oct 29 23:01:20.932372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.932344    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e3f789-5057-4149-adae-d626175767e0-plugins-config-volume" pod "f0e3f789-5057-4149-adae-d626175767e0" (UID: "f0e3f789-5057-4149-adae-d626175767e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:20.932895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.932868    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-plugins-config-volume" pod "bf0fa94f-5748-46f8-adeb-8ad243f3e72d" (UID: "bf0fa94f-5748-46f8-adeb-8ad243f3e72d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:20.934369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.934336    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e3f789-5057-4149-adae-d626175767e0-kube-api-access-hr7jl" pod "f0e3f789-5057-4149-adae-d626175767e0" (UID: "f0e3f789-5057-4149-adae-d626175767e0"). InnerVolumeSpecName "kube-api-access-hr7jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:20.934751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.934727    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-kube-api-access-jrqpg" pod "bf0fa94f-5748-46f8-adeb-8ad243f3e72d" (UID: "bf0fa94f-5748-46f8-adeb-8ad243f3e72d"). InnerVolumeSpecName "kube-api-access-jrqpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:20.972138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.972107    4827 generic.go:356] "Generic (PLEG): container finished" podID="bf0fa94f-5748-46f8-adeb-8ad243f3e72d" containerID="c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65" exitCode=0
Oct 29 23:01:20.972275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.972189    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"
Oct 29 23:01:20.972275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.972195    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl" event={"ID":"bf0fa94f-5748-46f8-adeb-8ad243f3e72d","Type":"ContainerDied","Data":"c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65"}
Oct 29 23:01:20.972386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.972288    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl" event={"ID":"bf0fa94f-5748-46f8-adeb-8ad243f3e72d","Type":"ContainerDied","Data":"a8558edc71c8682974eb87691b5d36cbdade311ecb37a219550ee1dfab685ef9"}
Oct 29 23:01:20.972386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.972322    4827 scope.go:122] "RemoveContainer" containerID="c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65"
Oct 29 23:01:20.973703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.973684    4827 generic.go:356] "Generic (PLEG): container finished" podID="f0e3f789-5057-4149-adae-d626175767e0" containerID="ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d" exitCode=0
Oct 29 23:01:20.973848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.973751    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"
Oct 29 23:01:20.973848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.973749    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx" event={"ID":"f0e3f789-5057-4149-adae-d626175767e0","Type":"ContainerDied","Data":"ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d"}
Oct 29 23:01:20.973960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.973875    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx" event={"ID":"f0e3f789-5057-4149-adae-d626175767e0","Type":"ContainerDied","Data":"4ff6b9ad74516d41cf9994b10e92b5018d10f7c28c9209fa6cebace1a828aa12"}
Oct 29 23:01:20.983953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.983935    4827 scope.go:122] "RemoveContainer" containerID="c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65"
Oct 29 23:01:20.984308 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:01:20.984285    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65\": container with ID starting with c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65 not found: ID does not exist" containerID="c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65"
Oct 29 23:01:20.984364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.984321    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65"} err="failed to get container status \"c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65\": rpc error: code = NotFound desc = could not find container \"c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65\": container with ID starting with c8f90d9d390622c384a51db584e69d91c836718e5ed270c012e78b6c09c64e65 not found: ID does not exist"
Oct 29 23:01:20.984364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.984345    4827 scope.go:122] "RemoveContainer" containerID="ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d"
Oct 29 23:01:20.994677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.994653    4827 scope.go:122] "RemoveContainer" containerID="ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d"
Oct 29 23:01:20.994985 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:01:20.994967    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d\": container with ID starting with ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d not found: ID does not exist" containerID="ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d"
Oct 29 23:01:20.995058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:20.994992    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d"} err="failed to get container status \"ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d\": rpc error: code = NotFound desc = could not find container \"ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d\": container with ID starting with ebf2b0badd0f62cb28771ae45febab99eaa567fcab134e6e1dcaf8ca4ec1ca4d not found: ID does not exist"
Oct 29 23:01:21.027721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.027696    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"]
Oct 29 23:01:21.032875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.032853    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-579cd7bf6d-l22xx"]
Oct 29 23:01:21.033272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.033246    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e3f789-5057-4149-adae-d626175767e0-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.033272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.033272    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrqpg\" (UniqueName: \"kubernetes.io/projected/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-kube-api-access-jrqpg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.033416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.033281    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0fa94f-5748-46f8-adeb-8ad243f3e72d-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.033416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.033290    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr7jl\" (UniqueName: \"kubernetes.io/projected/f0e3f789-5057-4149-adae-d626175767e0-kube-api-access-hr7jl\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.064715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.064686    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"]
Oct 29 23:01:21.068697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:21.068674    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-867cf6bd4c-7kdvl"]
Oct 29 23:01:22.536813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:22.521072    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf0fa94f-5748-46f8-adeb-8ad243f3e72d" path="/var/lib/kubelet/pods/bf0fa94f-5748-46f8-adeb-8ad243f3e72d/volumes"
Oct 29 23:01:22.536813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:22.521865    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e3f789-5057-4149-adae-d626175767e0" path="/var/lib/kubelet/pods/f0e3f789-5057-4149-adae-d626175767e0/volumes"
Oct 29 23:01:24.899582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.899554    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 23:01:24.970482 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970402    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-envoy\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.970482 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970439    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-credential-socket\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.970702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970487    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-socket\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.970702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970577    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-certs\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.970702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970665    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71fcbb11-1511-426f-9e8e-9052a7b55090-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71fcbb11-1511-426f-9e8e-9052a7b55090-istiod-ca-cert\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.970908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970708    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-kube-api-access-dxsgs\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-kube-api-access-dxsgs\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.970908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970826    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-credential-socket" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:24.970908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970843    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-data\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.970908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970857    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-socket" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:24.971132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970929    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-certs" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:24.971132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.970999    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-podinfo\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.971132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971062    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-token\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-token\") pod \"71fcbb11-1511-426f-9e8e-9052a7b55090\" (UID: \"71fcbb11-1511-426f-9e8e-9052a7b55090\") "
Oct 29 23:01:24.971132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971092    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-data" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:24.971132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971093    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fcbb11-1511-426f-9e8e-9052a7b55090-istiod-ca-cert" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:24.971851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971824    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.971986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971852    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.971986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971868    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.971986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971882    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71fcbb11-1511-426f-9e8e-9052a7b55090-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.971986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.971894    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.973019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.972996    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-envoy" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:24.973214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.973186    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-podinfo" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:24.973303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.973285    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-kube-api-access-dxsgs" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "kube-api-access-dxsgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:24.973574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:24.973557    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-token" pod "71fcbb11-1511-426f-9e8e-9052a7b55090" (UID: "71fcbb11-1511-426f-9e8e-9052a7b55090"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:25.070978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.070940    4827 generic.go:356] "Generic (PLEG): container finished" podID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerID="e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0" exitCode=0
Oct 29 23:01:25.071174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.070992    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"
Oct 29 23:01:25.071174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.071043    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" event={"ID":"71fcbb11-1511-426f-9e8e-9052a7b55090","Type":"ContainerDied","Data":"e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0"}
Oct 29 23:01:25.071174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.071118    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb" event={"ID":"71fcbb11-1511-426f-9e8e-9052a7b55090","Type":"ContainerDied","Data":"ad5fe27d46bc1fd63f7587d14270711cae126f32527d01a9252adb205fc8e15f"}
Oct 29 23:01:25.071174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.071151    4827 scope.go:122] "RemoveContainer" containerID="e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0"
Oct 29 23:01:25.073308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.073286    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.073457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.073313    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.073457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.073328    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71fcbb11-1511-426f-9e8e-9052a7b55090-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.073457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.073343    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxsgs\" (UniqueName: \"kubernetes.io/projected/71fcbb11-1511-426f-9e8e-9052a7b55090-kube-api-access-dxsgs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.095194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.095171    4827 scope.go:122] "RemoveContainer" containerID="2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3"
Oct 29 23:01:25.111707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.111661    4827 scope.go:122] "RemoveContainer" containerID="e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0"
Oct 29 23:01:25.112171 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:01:25.112149    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0\": container with ID starting with e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0 not found: ID does not exist" containerID="e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0"
Oct 29 23:01:25.112284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.112178    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0"} err="failed to get container status \"e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0\": rpc error: code = NotFound desc = could not find container \"e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0\": container with ID starting with e4bfe51af324d7b45fa25666d7224b2099f9be89a5f2b167957eb86dda4912e0 not found: ID does not exist"
Oct 29 23:01:25.112284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.112196    4827 scope.go:122] "RemoveContainer" containerID="2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3"
Oct 29 23:01:25.112530 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:01:25.112505    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3\": container with ID starting with 2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3 not found: ID does not exist" containerID="2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3"
Oct 29 23:01:25.112602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.112541    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3"} err="failed to get container status \"2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3\": rpc error: code = NotFound desc = could not find container \"2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3\": container with ID starting with 2ccf2390f38ac6efa60169b0ba8279c461a9910f808f679541a4538849008bf3 not found: ID does not exist"
Oct 29 23:01:25.127104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.126827    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"]
Oct 29 23:01:25.132240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:25.132217    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-qthfb"]
Oct 29 23:01:26.097837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.096873    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"]
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.097962    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.097985    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098004    4827 cpu_manager.go:414] "containerMap: removing container" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098012    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098035    4827 cpu_manager.go:414] "containerMap: removing container" podUID="bf0fa94f-5748-46f8-adeb-8ad243f3e72d" containerName="payload-pre-processing"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098046    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0fa94f-5748-46f8-adeb-8ad243f3e72d" containerName="payload-pre-processing"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098086    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f0e3f789-5057-4149-adae-d626175767e0" containerName="payload-processing"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098094    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e3f789-5057-4149-adae-d626175767e0" containerName="payload-processing"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098110    4827 cpu_manager.go:414] "containerMap: removing container" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098118    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098133    4827 cpu_manager.go:414] "containerMap: removing container" podUID="02a2c601-a75e-439a-9400-9ee9e86c8c6a" containerName="maas-api"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098140    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a2c601-a75e-439a-9400-9ee9e86c8c6a" containerName="maas-api"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098154    4827 cpu_manager.go:414] "containerMap: removing container" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098162    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098310    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0fa94f-5748-46f8-adeb-8ad243f3e72d" containerName="payload-pre-processing"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098331    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e3f789-5057-4149-adae-d626175767e0" containerName="payload-processing"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098344    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a2c601-a75e-439a-9400-9ee9e86c8c6a" containerName="maas-api"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098355    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098367    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d3629c-1694-4cb8-86ad-46803f41dfc0" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098381    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5f0ed7-ce14-4995-be27-a38b35058656" containerName="istio-proxy"
Oct 29 23:01:26.098441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.098393    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" containerName="istio-proxy"
Oct 29 23:01:26.104691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.104672    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"]
Oct 29 23:01:26.105893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.104801    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.133368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.133317    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0266dd-openshift-default-dockercfg-gr5jd\""
Oct 29 23:01:26.208849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.204685    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211390    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211486    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211540    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211572    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211601    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211646    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211745    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4cjm\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-kube-api-access-l4cjm\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.212351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.211844    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.314827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.314247    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.314827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.314454    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.314827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.314520    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.314827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.314557    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.315259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.314580    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.315354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.315340    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.315449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.315438    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.315582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.315567    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4cjm\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-kube-api-access-l4cjm\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.315720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.315709    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.315926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.315346    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.317231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.317179    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.319101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.317717    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.319101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.317885    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.319101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.317922    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.319393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.319174    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.324791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.323497    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.346686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.343198    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4cjm\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-kube-api-access-l4cjm\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.347567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.347541    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.428095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.428060    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:26.503078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.497543    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fcbb11-1511-426f-9e8e-9052a7b55090" path="/var/lib/kubelet/pods/71fcbb11-1511-426f-9e8e-9052a7b55090/volumes"
Oct 29 23:01:26.752739 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:01:26.752712    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb46a1dd_0383_4d4b_840e_6e45e6c76046.slice/crio-8cb8d0cf35adb1c949a33ac854cba82c7a4c595e9016f1c545eed7f47a0f6f36 WatchSource:0}: Error finding container 8cb8d0cf35adb1c949a33ac854cba82c7a4c595e9016f1c545eed7f47a0f6f36: Status 404 returned error can't find the container with id 8cb8d0cf35adb1c949a33ac854cba82c7a4c595e9016f1c545eed7f47a0f6f36
Oct 29 23:01:26.752901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.752739    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"]
Oct 29 23:01:26.755841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.755807    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:26.755947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.755876    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:26.755947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:26.755905    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:27.031309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:27.031180    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"]
Oct 29 23:01:27.035339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:27.035312    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b9b771-6vkdz"]
Oct 29 23:01:27.116582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:27.116545    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" event={"ID":"bb46a1dd-0383-4d4b-840e-6e45e6c76046","Type":"ContainerStarted","Data":"89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48"}
Oct 29 23:01:27.117027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:27.116603    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" event={"ID":"bb46a1dd-0383-4d4b-840e-6e45e6c76046","Type":"ContainerStarted","Data":"8cb8d0cf35adb1c949a33ac854cba82c7a4c595e9016f1c545eed7f47a0f6f36"}
Oct 29 23:01:27.169030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:27.168968    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podStartSLOduration=2.1689489 podStartE2EDuration="2.1689489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:27.165088591 +0000 UTC m=+1122.787999616" watchObservedRunningTime="2026-10-29 23:01:27.1689489 +0000 UTC m=+1122.791859921"
Oct 29 23:01:27.435118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:27.433232    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:27.573787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:27.573096    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:28.125382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:28.125351    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:28.126384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:28.126360    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:01:28.489684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:28.489603    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed" path="/var/lib/kubelet/pods/b8c4ef6a-a9af-4b33-9c43-e1286f8e31ed/volumes"
Oct 29 23:01:29.094570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.094540    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"]
Oct 29 23:01:29.100912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.100882    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.118981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.118951    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"]
Oct 29 23:01:29.189656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.189624    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:01:29.189656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.189641    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:01:29.208756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.208733    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.208885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.208850    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.209079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.209057    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lvv\" (UniqueName: \"kubernetes.io/projected/68672aca-f365-4f30-8771-b4b0e5955ad5-kube-api-access-x4lvv\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.311025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.310991    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lvv\" (UniqueName: \"kubernetes.io/projected/68672aca-f365-4f30-8771-b4b0e5955ad5-kube-api-access-x4lvv\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.311213 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.311158    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.311535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.311226    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.312021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.311999    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.313701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.313673    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.334171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.334152    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lvv\" (UniqueName: \"kubernetes.io/projected/68672aca-f365-4f30-8771-b4b0e5955ad5-kube-api-access-x4lvv\") pod \"maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.422233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.422191    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:29.814788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:29.814653    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"]
Oct 29 23:01:30.151092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:30.151048    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl" event={"ID":"68672aca-f365-4f30-8771-b4b0e5955ad5","Type":"ContainerStarted","Data":"cf02a3e84349bca8e17da3f006b4db51d1969bf7e0c9b059b7d81c131268179c"}
Oct 29 23:01:31.173176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.173140    4827 generic.go:356] "Generic (PLEG): container finished" podID="68672aca-f365-4f30-8771-b4b0e5955ad5" containerID="fccd26e6130785dc25de0c4d2c7a30a343168952f84721bf78376534c0ba67c8" exitCode=0
Oct 29 23:01:31.173667 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.173312    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl" event={"ID":"68672aca-f365-4f30-8771-b4b0e5955ad5","Type":"ContainerDied","Data":"fccd26e6130785dc25de0c4d2c7a30a343168952f84721bf78376534c0ba67c8"}
Oct 29 23:01:31.422923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.422894    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"]
Oct 29 23:01:31.427449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.427430    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:31.459032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.459002    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29ss\" (UniqueName: \"kubernetes.io/projected/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-kube-api-access-l29ss\") pod \"payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:31.459154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.459049    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:31.483242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.483100    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0266dd-dockercfg-jq4hc\""
Oct 29 23:01:31.484181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.484156    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0266dd\""
Oct 29 23:01:31.490997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.490972    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"]
Oct 29 23:01:31.562815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.561374    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29ss\" (UniqueName: \"kubernetes.io/projected/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-kube-api-access-l29ss\") pod \"payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:31.562815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.561474    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:31.562815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.562202    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:31.771842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:31.769226    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29ss\" (UniqueName: \"kubernetes.io/projected/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-kube-api-access-l29ss\") pod \"payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:32.069896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.044204    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:32.377372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.377342    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:32.421058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.421034    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"]
Oct 29 23:01:32.421509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.421476    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-xks-ca\") pod \"68672aca-f365-4f30-8771-b4b0e5955ad5\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") "
Oct 29 23:01:32.421621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.421607    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68672aca-f365-4f30-8771-b4b0e5955ad5-kube-api-access-x4lvv\" (UniqueName: \"kubernetes.io/projected/68672aca-f365-4f30-8771-b4b0e5955ad5-kube-api-access-x4lvv\") pod \"68672aca-f365-4f30-8771-b4b0e5955ad5\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") "
Oct 29 23:01:32.421673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.421639    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-openshift-ca\") pod \"68672aca-f365-4f30-8771-b4b0e5955ad5\" (UID: \"68672aca-f365-4f30-8771-b4b0e5955ad5\") "
Oct 29 23:01:32.422037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.422015    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-openshift-ca" pod "68672aca-f365-4f30-8771-b4b0e5955ad5" (UID: "68672aca-f365-4f30-8771-b4b0e5955ad5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:32.422428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.422408    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:32.422688 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:01:32.422664    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b8747f_33a6_4b68_9de1_e079a5ee7ac7.slice/crio-37acfe489729e6809ee4a0f64681a3f5e43be2c7048316985767f9eb9a32db7d WatchSource:0}: Error finding container 37acfe489729e6809ee4a0f64681a3f5e43be2c7048316985767f9eb9a32db7d: Status 404 returned error can't find the container with id 37acfe489729e6809ee4a0f64681a3f5e43be2c7048316985767f9eb9a32db7d
Oct 29 23:01:32.423658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.423557    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-xks-ca" pod "68672aca-f365-4f30-8771-b4b0e5955ad5" (UID: "68672aca-f365-4f30-8771-b4b0e5955ad5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:32.423839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.423817    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68672aca-f365-4f30-8771-b4b0e5955ad5-kube-api-access-x4lvv" pod "68672aca-f365-4f30-8771-b4b0e5955ad5" (UID: "68672aca-f365-4f30-8771-b4b0e5955ad5"). InnerVolumeSpecName "kube-api-access-x4lvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:32.524504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.524477    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68672aca-f365-4f30-8771-b4b0e5955ad5-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:32.524628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.524508    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4lvv\" (UniqueName: \"kubernetes.io/projected/68672aca-f365-4f30-8771-b4b0e5955ad5-kube-api-access-x4lvv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:32.592539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.592506    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"]
Oct 29 23:01:32.593340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.593289    4827 cpu_manager.go:414] "containerMap: removing container" podUID="68672aca-f365-4f30-8771-b4b0e5955ad5" containerName="revoke-keys"
Oct 29 23:01:32.593340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.593307    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68672aca-f365-4f30-8771-b4b0e5955ad5" containerName="revoke-keys"
Oct 29 23:01:32.593494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.593412    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="68672aca-f365-4f30-8771-b4b0e5955ad5" containerName="revoke-keys"
Oct 29 23:01:32.597109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.597094    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.619940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.619908    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"]
Oct 29 23:01:32.626233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.626201    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.626524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.626396    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9tq\" (UniqueName: \"kubernetes.io/projected/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-kube-api-access-kr9tq\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.626605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.626540    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.732806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.728813    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.732806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.729024    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9tq\" (UniqueName: \"kubernetes.io/projected/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-kube-api-access-kr9tq\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.732806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.729166    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.737845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.734548    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.760787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.753485    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.783690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.775606    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9tq\" (UniqueName: \"kubernetes.io/projected/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-kube-api-access-kr9tq\") pod \"maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:32.835110 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:01:32.835075    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:01:32.921091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:32.921059    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:33.194817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.194724    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw" event={"ID":"88b8747f-33a6-4b68-9de1-e079a5ee7ac7","Type":"ContainerStarted","Data":"f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4"}
Oct 29 23:01:33.194817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.194798    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw" event={"ID":"88b8747f-33a6-4b68-9de1-e079a5ee7ac7","Type":"ContainerStarted","Data":"37acfe489729e6809ee4a0f64681a3f5e43be2c7048316985767f9eb9a32db7d"}
Oct 29 23:01:33.195026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.194829    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:33.196543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.196515    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl" event={"ID":"68672aca-f365-4f30-8771-b4b0e5955ad5","Type":"ContainerDied","Data":"cf02a3e84349bca8e17da3f006b4db51d1969bf7e0c9b059b7d81c131268179c"}
Oct 29 23:01:33.196690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.196566    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf02a3e84349bca8e17da3f006b4db51d1969bf7e0c9b059b7d81c131268179c"
Oct 29 23:01:33.196690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.196608    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"
Oct 29 23:01:33.224933 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:01:33.224899    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110cd9ab_a6c4_40f0_a34d_3bbeafd58c5c.slice/crio-f16cb4cd57cdfd5222c89a52d895d0e43408875a009417c28028f230d8351081 WatchSource:0}: Error finding container f16cb4cd57cdfd5222c89a52d895d0e43408875a009417c28028f230d8351081: Status 404 returned error can't find the container with id f16cb4cd57cdfd5222c89a52d895d0e43408875a009417c28028f230d8351081
Oct 29 23:01:33.225363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.225345    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"]
Oct 29 23:01:33.240265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:33.240224    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw" podStartSLOduration=2.240209821 podStartE2EDuration="2.240209821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:33.234871423 +0000 UTC m=+1128.857782465" watchObservedRunningTime="2026-10-29 23:01:33.240209821 +0000 UTC m=+1128.863120878"
Oct 29 23:01:34.208014 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:34.207985    4827 generic.go:356] "Generic (PLEG): container finished" podID="110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" containerID="f51377169660622a7bc4dc7ceaf3eb5380153f7c0222886553dc5c201f7fc4ac" exitCode=0
Oct 29 23:01:34.208427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:34.208068    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77" event={"ID":"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c","Type":"ContainerDied","Data":"f51377169660622a7bc4dc7ceaf3eb5380153f7c0222886553dc5c201f7fc4ac"}
Oct 29 23:01:34.208427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:34.208119    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77" event={"ID":"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c","Type":"ContainerStarted","Data":"f16cb4cd57cdfd5222c89a52d895d0e43408875a009417c28028f230d8351081"}
Oct 29 23:01:35.379979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.379948    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:35.519805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.519671    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-openshift-ca\") pod \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") "
Oct 29 23:01:35.519805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.519732    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-kube-api-access-kr9tq\" (UniqueName: \"kubernetes.io/projected/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-kube-api-access-kr9tq\") pod \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") "
Oct 29 23:01:35.520095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.520050    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-openshift-ca" pod "110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" (UID: "110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:35.520095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.520061    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-xks-ca\") pod \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\" (UID: \"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c\") "
Oct 29 23:01:35.520725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.520701    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.521945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.521919    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-kube-api-access-kr9tq" pod "110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" (UID: "110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c"). InnerVolumeSpecName "kube-api-access-kr9tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:35.522060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.522018    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-xks-ca" pod "110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" (UID: "110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:35.622906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.622873    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.623054 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:35.622917    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr9tq\" (UniqueName: \"kubernetes.io/projected/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c-kube-api-access-kr9tq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:36.226182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:36.226148    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77" event={"ID":"110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c","Type":"ContainerDied","Data":"f16cb4cd57cdfd5222c89a52d895d0e43408875a009417c28028f230d8351081"}
Oct 29 23:01:36.226358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:36.226198    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f16cb4cd57cdfd5222c89a52d895d0e43408875a009417c28028f230d8351081"
Oct 29 23:01:36.226358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:36.226163    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"
Oct 29 23:01:40.974341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:40.974305    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"]
Oct 29 23:01:40.976049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:40.976006    4827 cpu_manager.go:414] "containerMap: removing container" podUID="110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" containerName="revoke-keys"
Oct 29 23:01:40.984852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:40.984817    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" containerName="revoke-keys"
Oct 29 23:01:40.985303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:40.985283    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" containerName="revoke-keys"
Oct 29 23:01:41.003606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.003574    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"]
Oct 29 23:01:41.003833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.003702    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.018508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.018486    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:01:41.030381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.030355    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:01:41.037956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.037933    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"]
Oct 29 23:01:41.044636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.044611    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7616a32-6qhqg"]
Oct 29 23:01:41.051320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.050905    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520dd7b8-c079-4339-9762-57adb965c776-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.051320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.050985    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520dd7b8-c079-4339-9762-57adb965c776-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.051320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.051055    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ndq\" (UniqueName: \"kubernetes.io/projected/520dd7b8-c079-4339-9762-57adb965c776-kube-api-access-f4ndq\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.152609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.152575    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520dd7b8-c079-4339-9762-57adb965c776-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.152826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.152626    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520dd7b8-c079-4339-9762-57adb965c776-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.152826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.152666    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ndq\" (UniqueName: \"kubernetes.io/projected/520dd7b8-c079-4339-9762-57adb965c776-kube-api-access-f4ndq\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.153351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.153326    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520dd7b8-c079-4339-9762-57adb965c776-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.155046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.155015    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520dd7b8-c079-4339-9762-57adb965c776-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.191508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.191439    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ndq\" (UniqueName: \"kubernetes.io/projected/520dd7b8-c079-4339-9762-57adb965c776-kube-api-access-f4ndq\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.322501 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.322422    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:41.724684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:41.724657    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"]
Oct 29 23:01:42.303109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:42.303072    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc" event={"ID":"520dd7b8-c079-4339-9762-57adb965c776","Type":"ContainerStarted","Data":"72a2ada16a4d90ec8fbef5ca9b1a882e9e3d98a6083aa8549e74240e3b96eabb"}
Oct 29 23:01:42.303529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:42.303141    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc" event={"ID":"520dd7b8-c079-4339-9762-57adb965c776","Type":"ContainerStarted","Data":"092d3710a9063b68310e80fcbd7f05815ac7424894fba168d121ed0ce14f13ff"}
Oct 29 23:01:42.347576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:42.347472    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc" podStartSLOduration=2.347458753 podStartE2EDuration="2.347458753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:42.342793717 +0000 UTC m=+1137.965704738" watchObservedRunningTime="2026-10-29 23:01:42.347458753 +0000 UTC m=+1137.970369797"
Oct 29 23:01:42.499207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:42.499174    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3" path="/var/lib/kubelet/pods/2c0d208f-c1ba-48e2-8fa1-9b51a9c651f3/volumes"
Oct 29 23:01:43.314685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:43.314651    4827 generic.go:356] "Generic (PLEG): container finished" podID="520dd7b8-c079-4339-9762-57adb965c776" containerID="72a2ada16a4d90ec8fbef5ca9b1a882e9e3d98a6083aa8549e74240e3b96eabb" exitCode=0
Oct 29 23:01:43.315128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:43.314742    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc" event={"ID":"520dd7b8-c079-4339-9762-57adb965c776","Type":"ContainerDied","Data":"72a2ada16a4d90ec8fbef5ca9b1a882e9e3d98a6083aa8549e74240e3b96eabb"}
Oct 29 23:01:44.210613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.210581    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:01:44.647538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.647504    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:44.719996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.719958    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520dd7b8-c079-4339-9762-57adb965c776-kube-api-access-f4ndq\" (UniqueName: \"kubernetes.io/projected/520dd7b8-c079-4339-9762-57adb965c776-kube-api-access-f4ndq\") pod \"520dd7b8-c079-4339-9762-57adb965c776\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") "
Oct 29 23:01:44.720219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.720059    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520dd7b8-c079-4339-9762-57adb965c776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520dd7b8-c079-4339-9762-57adb965c776-maas-api-openshift-ca\") pod \"520dd7b8-c079-4339-9762-57adb965c776\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") "
Oct 29 23:01:44.720219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.720144    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520dd7b8-c079-4339-9762-57adb965c776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520dd7b8-c079-4339-9762-57adb965c776-maas-api-xks-ca\") pod \"520dd7b8-c079-4339-9762-57adb965c776\" (UID: \"520dd7b8-c079-4339-9762-57adb965c776\") "
Oct 29 23:01:44.722788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.722745    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520dd7b8-c079-4339-9762-57adb965c776-maas-api-openshift-ca" pod "520dd7b8-c079-4339-9762-57adb965c776" (UID: "520dd7b8-c079-4339-9762-57adb965c776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:44.723649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.723623    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520dd7b8-c079-4339-9762-57adb965c776-maas-api-xks-ca" pod "520dd7b8-c079-4339-9762-57adb965c776" (UID: "520dd7b8-c079-4339-9762-57adb965c776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:44.725006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.724981    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520dd7b8-c079-4339-9762-57adb965c776-kube-api-access-f4ndq" pod "520dd7b8-c079-4339-9762-57adb965c776" (UID: "520dd7b8-c079-4339-9762-57adb965c776"). InnerVolumeSpecName "kube-api-access-f4ndq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:44.822405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.822365    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4ndq\" (UniqueName: \"kubernetes.io/projected/520dd7b8-c079-4339-9762-57adb965c776-kube-api-access-f4ndq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.822405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.822402    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520dd7b8-c079-4339-9762-57adb965c776-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.822595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:44.822415    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520dd7b8-c079-4339-9762-57adb965c776-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:01:45.750579 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.750550    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"]
Oct 29 23:01:45.751416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.751371    4827 cpu_manager.go:414] "containerMap: removing container" podUID="520dd7b8-c079-4339-9762-57adb965c776" containerName="revoke-keys"
Oct 29 23:01:45.751416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.751392    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520dd7b8-c079-4339-9762-57adb965c776" containerName="revoke-keys"
Oct 29 23:01:45.751600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.751499    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="520dd7b8-c079-4339-9762-57adb965c776" containerName="revoke-keys"
Oct 29 23:01:45.774927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.774890    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:45.801874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.778306    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc" event={"ID":"520dd7b8-c079-4339-9762-57adb965c776","Type":"ContainerDied","Data":"092d3710a9063b68310e80fcbd7f05815ac7424894fba168d121ed0ce14f13ff"}
Oct 29 23:01:45.801874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.778359    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092d3710a9063b68310e80fcbd7f05815ac7424894fba168d121ed0ce14f13ff"
Oct 29 23:01:45.801874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.778423    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"
Oct 29 23:01:45.817005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.803400    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f173b1-dockercfg-lrlf8\""
Oct 29 23:01:45.817005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.803511    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f173b1\""
Oct 29 23:01:45.817005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.814970    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"]
Oct 29 23:01:45.974985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.974947    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cb891c-a60e-4593-8b63-94dc2bd1924f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:45.975450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:45.975423    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24szf\" (UniqueName: \"kubernetes.io/projected/08cb891c-a60e-4593-8b63-94dc2bd1924f-kube-api-access-24szf\") pod \"payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:46.078845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.078410    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cb891c-a60e-4593-8b63-94dc2bd1924f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:46.078845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.078646    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24szf\" (UniqueName: \"kubernetes.io/projected/08cb891c-a60e-4593-8b63-94dc2bd1924f-kube-api-access-24szf\") pod \"payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:46.081797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.079943    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cb891c-a60e-4593-8b63-94dc2bd1924f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:46.144789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.140032    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24szf\" (UniqueName: \"kubernetes.io/projected/08cb891c-a60e-4593-8b63-94dc2bd1924f-kube-api-access-24szf\") pod \"payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:46.452794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.445255    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:46.661796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.661118    4827 scope.go:122] "RemoveContainer" containerID="c54c68f8b4a9cc666b89254c5a712c5ec9e8150a52199c461092d86cdc951460"
Oct 29 23:01:46.691923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.691710    4827 scope.go:122] "RemoveContainer" containerID="ff229a233e9bba8f66e121d9d1404e0e48a9f4a0b27fd54d9eff237842340c48"
Oct 29 23:01:46.715542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.715188    4827 scope.go:122] "RemoveContainer" containerID="74dfe5fa54920305b9f53a7f6ad480c4478b95ce942d8e1c80430c13c483a207"
Oct 29 23:01:46.730744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.730630    4827 scope.go:122] "RemoveContainer" containerID="553e43711ebd456fd5e7666336d4a16f4ddf5b46e9026a2c62ac72535a51fbfe"
Oct 29 23:01:46.741705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.741640    4827 scope.go:122] "RemoveContainer" containerID="05fa6288a902f8272c02d77325b69412d3cc7e005c63ee0a8fda53184b901623"
Oct 29 23:01:46.908490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.907266    4827 scope.go:122] "RemoveContainer" containerID="99fe289a9a1b6f75e9d089b0a1d141aa0142dfae927e4770c7dc8b4e87f2f16b"
Oct 29 23:01:46.963906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.963824    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"]
Oct 29 23:01:46.985960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:46.985150    4827 scope.go:122] "RemoveContainer" containerID="0d11f9fc5287dfdea29e3c00a5c1bfa3fe386412e5968e410400d6ac6adb73b5"
Oct 29 23:01:47.002967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:47.002941    4827 scope.go:122] "RemoveContainer" containerID="2ae1907c5cea303c6451fa9963e2379e63d526e72241c8a0499f5da0a5d8a976"
Oct 29 23:01:47.017149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:47.017051    4827 scope.go:122] "RemoveContainer" containerID="7b720d98aaa995aadbbe75c934f62bc85be4923e3f26a6c65ee18b5c6a456eb3"
Oct 29 23:01:48.058787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:48.049259    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw" event={"ID":"08cb891c-a60e-4593-8b63-94dc2bd1924f","Type":"ContainerStarted","Data":"d11c93dfd0306a31f75f7906402779f40fa7f9658c7339454d4833dc3102515f"}
Oct 29 23:01:49.032625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:49.032588    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"]
Oct 29 23:01:49.041882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:49.041853    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3413b987-46s9m"]
Oct 29 23:01:50.501345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:50.501301    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f75a5ce-e2a9-407a-b399-937dd1765358" path="/var/lib/kubelet/pods/5f75a5ce-e2a9-407a-b399-937dd1765358/volumes"
Oct 29 23:01:51.166954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:51.165815    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw" event={"ID":"08cb891c-a60e-4593-8b63-94dc2bd1924f","Type":"ContainerStarted","Data":"1ae77672410e4a707dd6ed4e9867fd59574591807504684a0662db77d2a66e69"}
Oct 29 23:01:51.166954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:51.166579    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:01:51.305339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:51.305041    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw" podStartSLOduration=3.244477107 podStartE2EDuration="6.305024536s" totalImagesPullingTime="3.060547429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:51.234700642 +0000 UTC m=+1146.857611661" watchObservedRunningTime="2026-10-29 23:01:51.305024536 +0000 UTC m=+1146.927935933"
Oct 29 23:01:53.215954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:53.215881    4827 generic.go:356] "Generic (PLEG): container finished" podID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerID="2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271" exitCode=0
Oct 29 23:01:53.215954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:53.215925    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" event={"ID":"0bb6d96c-910a-416e-aa2b-87bbf273de36","Type":"ContainerDied","Data":"2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271"}
Oct 29 23:01:53.216479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:53.216425    4827 scope.go:122] "RemoveContainer" containerID="2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271"
Oct 29 23:01:53.218214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:53.218179    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:53.218319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:53.218250    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:53.218319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:53.218283    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:54.227508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:54.227470    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" event={"ID":"0bb6d96c-910a-416e-aa2b-87bbf273de36","Type":"ContainerStarted","Data":"688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57"}
Oct 29 23:01:54.691418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:54.691379    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 23:01:54.691845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:54.691814    4827 patch_prober.go:29] interesting pod/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:01:54.691980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:54.691867    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:01:55.694243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:55.694202    4827 patch_prober.go:29] interesting pod/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:01:55.694894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:55.694862    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:01:55.884196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:55.884159    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:01:55.884460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:55.884435    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:01:56.160604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:56.160556    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"]
Oct 29 23:01:56.166323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:56.166298    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-36befd-6ll77"]
Oct 29 23:01:56.540842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:56.540196    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c" path="/var/lib/kubelet/pods/110cd9ab-a6c4-40f0-a34d-3bbeafd58c5c/volumes"
Oct 29 23:01:56.692347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:56.691973    4827 patch_prober.go:29] interesting pod/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:01:56.692347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:56.692045    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:01:57.692840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:57.692393    4827 patch_prober.go:29] interesting pod/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:01:57.692840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:57.692452    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:01:58.693221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:58.693176    4827 patch_prober.go:29] interesting pod/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:01:58.693847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:58.693813    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:01:59.329349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.329326    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 23:01:59.407963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.407881    4827 generic.go:356] "Generic (PLEG): container finished" podID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerID="56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118" exitCode=0
Oct 29 23:01:59.408118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.407954    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" event={"ID":"bbd3d0b2-dc85-4627-a891-0c5d33639748","Type":"ContainerDied","Data":"56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118"}
Oct 29 23:01:59.408495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.408466    4827 scope.go:122] "RemoveContainer" containerID="56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118"
Oct 29 23:01:59.410232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.410206    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:59.410301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.410276    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:59.410342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.410308    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:01:59.691733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.691698    4827 patch_prober.go:29] interesting pod/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:01:59.692238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.692210    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:01:59.757778 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.757735    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"]
Oct 29 23:01:59.762379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.762358    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.778225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.777900    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0266dd-serving-cert\""
Oct 29 23:01:59.787067 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.787039    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"]
Oct 29 23:01:59.857049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.857013    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.857225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.857194    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrgm\" (UniqueName: \"kubernetes.io/projected/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-kube-api-access-pvrgm\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.857328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.857312    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-metrics-certs\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.857477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.857457    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.959833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.958925    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.959833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.959076    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.959833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.959195    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrgm\" (UniqueName: \"kubernetes.io/projected/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-kube-api-access-pvrgm\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.959833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.959302    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-metrics-certs\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.961677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.960390    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:01:59.963032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:01:59.962991    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-metrics-certs\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:02:00.018795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:00.018538    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrgm\" (UniqueName: \"kubernetes.io/projected/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-kube-api-access-pvrgm\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:02:00.029169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:00.019006    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:02:00.095788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:00.095133    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:02:00.418349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:00.418309    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" event={"ID":"bbd3d0b2-dc85-4627-a891-0c5d33639748","Type":"ContainerStarted","Data":"cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0"}
Oct 29 23:02:00.525419 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:02:00.525387    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335e3b92_663b_4c2d_9efa_77bea4a7fa1d.slice/crio-1de4a63bc24183401297b253d5ca3c6b26e3e2fa1c06674ffa7df3646bac5321 WatchSource:0}: Error finding container 1de4a63bc24183401297b253d5ca3c6b26e3e2fa1c06674ffa7df3646bac5321: Status 404 returned error can't find the container with id 1de4a63bc24183401297b253d5ca3c6b26e3e2fa1c06674ffa7df3646bac5321
Oct 29 23:02:00.526128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:00.526107    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"]
Oct 29 23:02:00.702177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:00.702149    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 23:02:00.704212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:00.704190    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 23:02:01.289023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.288984    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:02:01.353478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.289585    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 23:02:01.353478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.352677    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 23:02:01.431733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.431701    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b" event={"ID":"335e3b92-663b-4c2d-9efa-77bea4a7fa1d","Type":"ContainerStarted","Data":"1de4a63bc24183401297b253d5ca3c6b26e3e2fa1c06674ffa7df3646bac5321"}
Oct 29 23:02:01.431877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.431812    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:02:01.454027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.453990    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"]
Oct 29 23:02:01.454244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.454218    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9" podUID="e8bf8095-b37f-4d76-9572-96e1cb459a80" containerName="maas-api" containerID="cri-o://22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa" gracePeriod=30
Oct 29 23:02:01.497610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.497020    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b" podStartSLOduration=1.848027184 podStartE2EDuration="2.496997623s" totalImagesPullingTime="648.970439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:01.485352754 +0000 UTC m=+1157.108263813" watchObservedRunningTime="2026-10-29 23:02:01.496997623 +0000 UTC m=+1157.119908646"
Oct 29 23:02:02.004865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:01.987604    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 23:02:02.126790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.125725    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bf8095-b37f-4d76-9572-96e1cb459a80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf8095-b37f-4d76-9572-96e1cb459a80-openshift-service-ca\") pod \"e8bf8095-b37f-4d76-9572-96e1cb459a80\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") "
Oct 29 23:02:02.126790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.126167    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bf8095-b37f-4d76-9572-96e1cb459a80-kube-api-access-vhqs6\" (UniqueName: \"kubernetes.io/projected/e8bf8095-b37f-4d76-9572-96e1cb459a80-kube-api-access-vhqs6\") pod \"e8bf8095-b37f-4d76-9572-96e1cb459a80\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") "
Oct 29 23:02:02.126790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.126368    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-metrics-certs\") pod \"e8bf8095-b37f-4d76-9572-96e1cb459a80\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") "
Oct 29 23:02:02.126790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.126499    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-maas-api-tls\") pod \"e8bf8095-b37f-4d76-9572-96e1cb459a80\" (UID: \"e8bf8095-b37f-4d76-9572-96e1cb459a80\") "
Oct 29 23:02:02.135788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.132972    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bf8095-b37f-4d76-9572-96e1cb459a80-openshift-service-ca" pod "e8bf8095-b37f-4d76-9572-96e1cb459a80" (UID: "e8bf8095-b37f-4d76-9572-96e1cb459a80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:02.155814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.150635    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf8095-b37f-4d76-9572-96e1cb459a80-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:02.174552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.174514    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-metrics-certs" pod "e8bf8095-b37f-4d76-9572-96e1cb459a80" (UID: "e8bf8095-b37f-4d76-9572-96e1cb459a80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:02.185856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.185815    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-maas-api-tls" pod "e8bf8095-b37f-4d76-9572-96e1cb459a80" (UID: "e8bf8095-b37f-4d76-9572-96e1cb459a80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:02.355480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.241983    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:02:02.355480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.273411    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bf8095-b37f-4d76-9572-96e1cb459a80-kube-api-access-vhqs6" pod "e8bf8095-b37f-4d76-9572-96e1cb459a80" (UID: "e8bf8095-b37f-4d76-9572-96e1cb459a80"). InnerVolumeSpecName "kube-api-access-vhqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:02.355480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.353163    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 23:02:02.355480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.353214    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 23:02:02.398654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.396478    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhqs6\" (UniqueName: \"kubernetes.io/projected/e8bf8095-b37f-4d76-9572-96e1cb459a80-kube-api-access-vhqs6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:02.398654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.396519    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:02.398654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.396536    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8bf8095-b37f-4d76-9572-96e1cb459a80-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:02.677894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.596050    4827 generic.go:356] "Generic (PLEG): container finished" podID="e8bf8095-b37f-4d76-9572-96e1cb459a80" containerID="22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa" exitCode=0
Oct 29 23:02:02.677894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.596150    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9" event={"ID":"e8bf8095-b37f-4d76-9572-96e1cb459a80","Type":"ContainerDied","Data":"22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa"}
Oct 29 23:02:02.677894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.596207    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9" event={"ID":"e8bf8095-b37f-4d76-9572-96e1cb459a80","Type":"ContainerDied","Data":"c1289fa1f24d3c621b4c692627194ea627650ab6b13c0bb796343a106bf3d7dd"}
Oct 29 23:02:02.677894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.596239    4827 scope.go:122] "RemoveContainer" containerID="22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa"
Oct 29 23:02:02.677894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.596405    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"
Oct 29 23:02:02.677894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.604661    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b" event={"ID":"335e3b92-663b-4c2d-9efa-77bea4a7fa1d","Type":"ContainerStarted","Data":"8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6"}
Oct 29 23:02:02.713061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.713008    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"]
Oct 29 23:02:02.715916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.715867    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1d1da-6qxrc"]
Oct 29 23:02:02.726211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.726131    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"]
Oct 29 23:02:02.730571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.730519    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1d1da-56b57c9574-g5xn9"]
Oct 29 23:02:02.736859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.736833    4827 scope.go:122] "RemoveContainer" containerID="22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa"
Oct 29 23:02:02.738374 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:02.738305    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa\": container with ID starting with 22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa not found: ID does not exist" containerID="22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa"
Oct 29 23:02:02.738374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:02.738347    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa"} err="failed to get container status \"22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa\": rpc error: code = NotFound desc = could not find container \"22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa\": container with ID starting with 22c91e249fb80fa5545a43a374d7c3a73eb2204111e40f45177a71d18c30c2fa not found: ID does not exist"
Oct 29 23:02:02.900290 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:02.900255    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:02:03.285790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.285183    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 23:02:03.285790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.285246    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 23:02:03.614972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.614880    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"]
Oct 29 23:02:03.615743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.615723    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e8bf8095-b37f-4d76-9572-96e1cb459a80" containerName="maas-api"
Oct 29 23:02:03.615743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.615743    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bf8095-b37f-4d76-9572-96e1cb459a80" containerName="maas-api"
Oct 29 23:02:03.615977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.615963    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bf8095-b37f-4d76-9572-96e1cb459a80" containerName="maas-api"
Oct 29 23:02:03.618364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.618346    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.644112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.644088    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"]
Oct 29 23:02:03.647923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.647903    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-848fb8-openshift-default-dockercfg-2ld7k\""
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.728917    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istiod-ca-cert\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729006    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-certs\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729145    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-podinfo\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729255    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-credential-socket\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729358    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-envoy\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729388    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzmm\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-kube-api-access-5qzmm\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729456    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-socket\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729483    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-token\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.729797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.729540    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-data\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.879845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.868461    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-podinfo\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.879845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.868586    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-credential-socket\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.879845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.868729    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-envoy\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.947094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.883366    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-credential-socket\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:03.992791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:03.976612    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-envoy\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.014802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.011867    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzmm\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-kube-api-access-5qzmm\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.049165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.040119    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzmm\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-kube-api-access-5qzmm\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.055789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.054468    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-socket\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.055789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.054521    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-token\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.055789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.054574    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-data\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.055789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.054656    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istiod-ca-cert\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.055789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.054734    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-certs\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.070788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.056423    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-socket\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.070788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.056674    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-data\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.070788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.062392    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istiod-ca-cert\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.218849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.209479    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-podinfo\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.218849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.209584    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-certs\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.241329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.241282    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-token\") pod \"e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") " pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.305828 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.297221    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 23:02:04.305828 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.297290    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 23:02:04.537086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.536992    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:04.544788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.542885    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"]
Oct 29 23:02:04.555522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.555405    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" containerID="cri-o://2c42d9deda369dfd59b9612ac3e0b1f84a052f57f1570d2cff1e62bffcd3d819" gracePeriod=30
Oct 29 23:02:04.559560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.559534    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520dd7b8-c079-4339-9762-57adb965c776" path="/var/lib/kubelet/pods/520dd7b8-c079-4339-9762-57adb965c776/volumes"
Oct 29 23:02:04.560324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:04.560302    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bf8095-b37f-4d76-9572-96e1cb459a80" path="/var/lib/kubelet/pods/e8bf8095-b37f-4d76-9572-96e1cb459a80/volumes"
Oct 29 23:02:05.138804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.135588    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"]
Oct 29 23:02:05.146312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.142530    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e1dc368-mfjc5"]
Oct 29 23:02:05.150611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.147496    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"]
Oct 29 23:02:05.156202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.153756    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e2a1a26-wwqq6"]
Oct 29 23:02:05.286859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.286287    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 23:02:05.286859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.286344    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 23:02:05.394312 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:02:05.394271    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e8c9af_5148_4b43_85ea_8aa3c4f81d1d.slice/crio-0930fac91c509c48c69935c7cdb4e0ebb02ad309f9cd7c697680ceac55417335 WatchSource:0}: Error finding container 0930fac91c509c48c69935c7cdb4e0ebb02ad309f9cd7c697680ceac55417335: Status 404 returned error can't find the container with id 0930fac91c509c48c69935c7cdb4e0ebb02ad309f9cd7c697680ceac55417335
Oct 29 23:02:05.395273 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.395246    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"]
Oct 29 23:02:05.397291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.397259    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:05.397365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.397325    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:05.397365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.397356    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:05.732140 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.732016    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" event={"ID":"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d","Type":"ContainerStarted","Data":"8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604"}
Oct 29 23:02:05.733284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:05.733258    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" event={"ID":"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d","Type":"ContainerStarted","Data":"0930fac91c509c48c69935c7cdb4e0ebb02ad309f9cd7c697680ceac55417335"}
Oct 29 23:02:06.292258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.292222    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 23:02:06.312790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.296915    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 23:02:06.580854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.580192    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd433c4-f254-4087-a651-74ef9cc12d96" path="/var/lib/kubelet/pods/1dd433c4-f254-4087-a651-74ef9cc12d96/volumes"
Oct 29 23:02:06.581025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.581009    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7304cb6-90c9-45c0-b6a4-5caec95733a6" path="/var/lib/kubelet/pods/a7304cb6-90c9-45c0-b6a4-5caec95733a6/volumes"
Oct 29 23:02:06.594351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.581519    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:06.594351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.581635    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:06.695501 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.695131    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" podStartSLOduration=3.69511196 podStartE2EDuration="3.69511196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:05.784117073 +0000 UTC m=+1161.407028100" watchObservedRunningTime="2026-10-29 23:02:06.69511196 +0000 UTC m=+1162.318022999"
Oct 29 23:02:06.743284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.743251    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:06.744320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:06.744294    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:02:07.284785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:07.284741    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:02:07.285164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:07.285144    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:02:07.285884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:07.285859    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:02:08.633209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:08.633178    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:02:09.782150 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.781795    4827 generic.go:356] "Generic (PLEG): container finished" podID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerID="2c42d9deda369dfd59b9612ac3e0b1f84a052f57f1570d2cff1e62bffcd3d819" exitCode=0
Oct 29 23:02:09.782150 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.781976    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" event={"ID":"8b985efd-ab05-4960-ae31-13c61bf74b1b","Type":"ContainerDied","Data":"2c42d9deda369dfd59b9612ac3e0b1f84a052f57f1570d2cff1e62bffcd3d819"}
Oct 29 23:02:09.782150 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.782054    4827 scope.go:122] "RemoveContainer" containerID="475fd2bdd94f3d24ba1d52b3c50bb0312f38b15ad30df1884b58b42090e204e8"
Oct 29 23:02:09.912750 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.912716    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 23:02:09.996673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.996637    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-podinfo\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.996917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.996701    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-credential-socket\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.996917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.996777    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-data\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.996917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.996808    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b985efd-ab05-4960-ae31-13c61bf74b1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b985efd-ab05-4960-ae31-13c61bf74b1b-istiod-ca-cert\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.996917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.996880    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-kube-api-access-blmbp\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-kube-api-access-blmbp\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.997111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.996945    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-envoy\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.997111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.996975    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-token\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-token\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.997111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.997024    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-socket\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.997111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.997045    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-certs\") pod \"8b985efd-ab05-4960-ae31-13c61bf74b1b\" (UID: \"8b985efd-ab05-4960-ae31-13c61bf74b1b\") "
Oct 29 23:02:09.998048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.997821    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-certs" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:09.998741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.998712    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-credential-socket" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:09.998994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.998969    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-data" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:09.999203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.999178    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-socket" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:09.999410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:09.999360    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b985efd-ab05-4960-ae31-13c61bf74b1b-istiod-ca-cert" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:10.001048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.000990    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-token" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:10.001048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.001025    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-kube-api-access-blmbp" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "kube-api-access-blmbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:10.001212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.001069    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-podinfo" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:10.001713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.001690    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-envoy" pod "8b985efd-ab05-4960-ae31-13c61bf74b1b" (UID: "8b985efd-ab05-4960-ae31-13c61bf74b1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.099515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099471    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099514    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099530    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099544    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b985efd-ab05-4960-ae31-13c61bf74b1b-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099557    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmbp\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-kube-api-access-blmbp\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099571    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099583    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b985efd-ab05-4960-ae31-13c61bf74b1b-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099596    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.099712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.099609    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b985efd-ab05-4960-ae31-13c61bf74b1b-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.142425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.142345    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"]
Oct 29 23:02:10.142610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.142578    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy" containerID="cri-o://f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a" gracePeriod=30
Oct 29 23:02:10.795644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.795551    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"
Oct 29 23:02:10.795644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.795568    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" event={"ID":"8b985efd-ab05-4960-ae31-13c61bf74b1b","Type":"ContainerDied","Data":"cee1db23b0d2b1e843d3dc6a0f0490894b664f2cb059b21cd8e19a168fa8788e"}
Oct 29 23:02:10.795644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.795635    4827 scope.go:122] "RemoveContainer" containerID="2c42d9deda369dfd59b9612ac3e0b1f84a052f57f1570d2cff1e62bffcd3d819"
Oct 29 23:02:10.850185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.850155    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"]
Oct 29 23:02:10.861865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.861840    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh"]
Oct 29 23:02:10.887061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.887027    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:02:10.887184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:10.887096    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-36befd-openshift-default-6fd4bd787f-kcnhh" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:12.491207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:12.491166    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" path="/var/lib/kubelet/pods/8b985efd-ab05-4960-ae31-13c61bf74b1b/volumes"
Oct 29 23:02:13.029710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:13.029678    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"]
Oct 29 23:02:13.041419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:13.041392    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade9c476-6b4sl"]
Oct 29 23:02:14.490818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:14.490634    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb" path="/var/lib/kubelet/pods/2e439c78-2880-4ea8-9bfb-bdd0fe5efaeb/volumes"
Oct 29 23:02:15.438266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.438231    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 23:02:15.554904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.554875    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-data\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.554923    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-envoy\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.554977    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-credential-socket\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555034    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-kube-api-access-kjlvg\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-kube-api-access-kjlvg\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555061    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-certs\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555100    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-podinfo\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555143    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-socket\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555207    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-data" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555222    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79b06ae-bc79-45a0-acec-9462a314fba1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79b06ae-bc79-45a0-acec-9462a314fba1-istiod-ca-cert\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555293    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-credential-socket" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:15.555932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555518    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-socket" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:15.555932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555567    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-token\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-token\") pod \"a79b06ae-bc79-45a0-acec-9462a314fba1\" (UID: \"a79b06ae-bc79-45a0-acec-9462a314fba1\") "
Oct 29 23:02:15.555932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555866    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79b06ae-bc79-45a0-acec-9462a314fba1-istiod-ca-cert" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:15.556113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.555957    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-certs" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:15.556699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.556606    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.556699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.556631    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.556699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.556646    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.556699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.556660    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79b06ae-bc79-45a0-acec-9462a314fba1-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.556699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.556672    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.558461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.558432    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-token" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:15.559456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.559431    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-envoy" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:15.559758 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.559733    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-podinfo" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:15.561129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.561103    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-kube-api-access-kjlvg" pod "a79b06ae-bc79-45a0-acec-9462a314fba1" (UID: "a79b06ae-bc79-45a0-acec-9462a314fba1"). InnerVolumeSpecName "kube-api-access-kjlvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:15.658377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.658345    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.658377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.658374    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjlvg\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-kube-api-access-kjlvg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.658377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.658385    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.658597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.658394    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79b06ae-bc79-45a0-acec-9462a314fba1-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:15.839095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.838999    4827 generic.go:356] "Generic (PLEG): container finished" podID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerID="f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a" exitCode=0
Oct 29 23:02:15.839095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.839052    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"
Oct 29 23:02:15.839095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.839074    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" event={"ID":"a79b06ae-bc79-45a0-acec-9462a314fba1","Type":"ContainerDied","Data":"f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a"}
Oct 29 23:02:15.839379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.839121    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg" event={"ID":"a79b06ae-bc79-45a0-acec-9462a314fba1","Type":"ContainerDied","Data":"6f92bce815da7a94ad622961672a4fb7c218becbcbb67894fe0a9f7e9ef8cd48"}
Oct 29 23:02:15.839379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.839146    4827 scope.go:122] "RemoveContainer" containerID="f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a"
Oct 29 23:02:15.851458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.851442    4827 scope.go:122] "RemoveContainer" containerID="b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44"
Oct 29 23:02:15.861947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.861929    4827 scope.go:122] "RemoveContainer" containerID="f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a"
Oct 29 23:02:15.862240 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:15.862221    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a\": container with ID starting with f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a not found: ID does not exist" containerID="f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a"
Oct 29 23:02:15.862320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.862249    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a"} err="failed to get container status \"f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a\": rpc error: code = NotFound desc = could not find container \"f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a\": container with ID starting with f46cb91cb9c06cdf4210c3ca3c54d69e49b39a7ed9c85ff341e930453805c90a not found: ID does not exist"
Oct 29 23:02:15.862320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.862267    4827 scope.go:122] "RemoveContainer" containerID="b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44"
Oct 29 23:02:15.862548 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:15.862532    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44\": container with ID starting with b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44 not found: ID does not exist" containerID="b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44"
Oct 29 23:02:15.862599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.862553    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44"} err="failed to get container status \"b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44\": rpc error: code = NotFound desc = could not find container \"b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44\": container with ID starting with b5d5a6d1646c93463419de722d0e741b21dcda6797210dd3f2cb43a65c82cf44 not found: ID does not exist"
Oct 29 23:02:15.892939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.892915    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"]
Oct 29 23:02:15.905807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:15.905752    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1d1da-openshift-default-b86f567f9-dhmfg"]
Oct 29 23:02:16.489383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:16.489353    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" path="/var/lib/kubelet/pods/a79b06ae-bc79-45a0-acec-9462a314fba1/volumes"
Oct 29 23:02:22.657219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:22.644598    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"]
Oct 29 23:02:22.657219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:22.644812    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx" podUID="1a435022-535c-480e-b06b-d6da1508eb97" containerName="maas-api" containerID="cri-o://c8012efaed1ef503bdd1d66320bca8e6403d95ce168f7ce7ea91c4a49e5efcd6" gracePeriod=30
Oct 29 23:02:22.825088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:22.825056    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"]
Oct 29 23:02:22.829482 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:22.829452    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-49dbb5-pgkfl"]
Oct 29 23:02:22.932385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:22.932309    4827 generic.go:356] "Generic (PLEG): container finished" podID="1a435022-535c-480e-b06b-d6da1508eb97" containerID="c8012efaed1ef503bdd1d66320bca8e6403d95ce168f7ce7ea91c4a49e5efcd6" exitCode=0
Oct 29 23:02:22.932517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:22.932387    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx" event={"ID":"1a435022-535c-480e-b06b-d6da1508eb97","Type":"ContainerDied","Data":"c8012efaed1ef503bdd1d66320bca8e6403d95ce168f7ce7ea91c4a49e5efcd6"}
Oct 29 23:02:23.314312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.314282    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:02:23.459519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.457807    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a435022-535c-480e-b06b-d6da1508eb97-kube-api-access-4crgj\" (UniqueName: \"kubernetes.io/projected/1a435022-535c-480e-b06b-d6da1508eb97-kube-api-access-4crgj\") pod \"1a435022-535c-480e-b06b-d6da1508eb97\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") "
Oct 29 23:02:23.459519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.457865    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-metrics-certs\") pod \"1a435022-535c-480e-b06b-d6da1508eb97\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") "
Oct 29 23:02:23.465928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.464702    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a435022-535c-480e-b06b-d6da1508eb97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a435022-535c-480e-b06b-d6da1508eb97-openshift-service-ca\") pod \"1a435022-535c-480e-b06b-d6da1508eb97\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") "
Oct 29 23:02:23.465928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.464778    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-maas-api-tls\") pod \"1a435022-535c-480e-b06b-d6da1508eb97\" (UID: \"1a435022-535c-480e-b06b-d6da1508eb97\") "
Oct 29 23:02:23.466741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.466693    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a435022-535c-480e-b06b-d6da1508eb97-openshift-service-ca" pod "1a435022-535c-480e-b06b-d6da1508eb97" (UID: "1a435022-535c-480e-b06b-d6da1508eb97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:23.500609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.499333    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a435022-535c-480e-b06b-d6da1508eb97-kube-api-access-4crgj" pod "1a435022-535c-480e-b06b-d6da1508eb97" (UID: "1a435022-535c-480e-b06b-d6da1508eb97"). InnerVolumeSpecName "kube-api-access-4crgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:23.530852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.524795    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-maas-api-tls" pod "1a435022-535c-480e-b06b-d6da1508eb97" (UID: "1a435022-535c-480e-b06b-d6da1508eb97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:23.538692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.538636    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-metrics-certs" pod "1a435022-535c-480e-b06b-d6da1508eb97" (UID: "1a435022-535c-480e-b06b-d6da1508eb97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:23.568062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.567476    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a435022-535c-480e-b06b-d6da1508eb97-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:23.568062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.567517    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:23.568062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.567526    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4crgj\" (UniqueName: \"kubernetes.io/projected/1a435022-535c-480e-b06b-d6da1508eb97-kube-api-access-4crgj\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:23.568062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.567534    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a435022-535c-480e-b06b-d6da1508eb97-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:23.951416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.951381    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"
Oct 29 23:02:23.951416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.951397    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx" event={"ID":"1a435022-535c-480e-b06b-d6da1508eb97","Type":"ContainerDied","Data":"9feb44314e869735982b840ad7e8a2b36b738fc723c61c6021c1e55ae08e0a8d"}
Oct 29 23:02:23.952420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.951463    4827 scope.go:122] "RemoveContainer" containerID="c8012efaed1ef503bdd1d66320bca8e6403d95ce168f7ce7ea91c4a49e5efcd6"
Oct 29 23:02:23.988474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.988442    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"]
Oct 29 23:02:23.988663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:23.988642    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy" containerID="cri-o://0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75" gracePeriod=30
Oct 29 23:02:24.017270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:24.017186    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"]
Oct 29 23:02:24.028289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:24.028256    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-13aa41-75d9bbb854-7mmnx"]
Oct 29 23:02:24.502246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:24.502204    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a435022-535c-480e-b06b-d6da1508eb97" path="/var/lib/kubelet/pods/1a435022-535c-480e-b06b-d6da1508eb97/volumes"
Oct 29 23:02:24.502655 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:24.502638    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68672aca-f365-4f30-8771-b4b0e5955ad5" path="/var/lib/kubelet/pods/68672aca-f365-4f30-8771-b4b0e5955ad5/volumes"
Oct 29 23:02:27.260087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:27.260051    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"]
Oct 29 23:02:27.260558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:27.260261    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" containerID="cri-o://385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad" gracePeriod=30
Oct 29 23:02:29.740381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.740301    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 23:02:29.824898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.824868    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-kube-api-access-fzxjc\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-kube-api-access-fzxjc\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.824928    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-certs\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.824961    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-token\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-token\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825032    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4025821f-f08b-41af-9cd0-45a8b54f5d78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4025821f-f08b-41af-9cd0-45a8b54f5d78-istiod-ca-cert\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825125    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-data\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825250    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-envoy\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825311    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-credential-socket\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825361    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-socket\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.825630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825597    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4025821f-f08b-41af-9cd0-45a8b54f5d78-istiod-ca-cert" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:29.826107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825792    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-podinfo\") pod \"4025821f-f08b-41af-9cd0-45a8b54f5d78\" (UID: \"4025821f-f08b-41af-9cd0-45a8b54f5d78\") "
Oct 29 23:02:29.826758 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.826728    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-credential-socket" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:29.826884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.826146    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-socket" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:29.826884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.825915    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-data" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:29.827542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.827027    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.827542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.827046    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.827542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.827063    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4025821f-f08b-41af-9cd0-45a8b54f5d78-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.827542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.827078    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.827542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.827310    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-certs" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:29.828223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.828197    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-kube-api-access-fzxjc" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "kube-api-access-fzxjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:29.829049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.829024    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-token" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:29.829553 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.829532    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-podinfo" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:29.831016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.830989    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-envoy" pod "4025821f-f08b-41af-9cd0-45a8b54f5d78" (UID: "4025821f-f08b-41af-9cd0-45a8b54f5d78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:29.928141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.928112    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.928141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.928138    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.928141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.928149    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxjc\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-kube-api-access-fzxjc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.928417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.928161    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4025821f-f08b-41af-9cd0-45a8b54f5d78-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.928417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:29.928170    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4025821f-f08b-41af-9cd0-45a8b54f5d78-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:30.110716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.110634    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"]
Oct 29 23:02:30.111223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.111170    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" containerID="cri-o://688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57" gracePeriod=30
Oct 29 23:02:30.161833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.147871    4827 generic.go:356] "Generic (PLEG): container finished" podID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerID="0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75" exitCode=0
Oct 29 23:02:30.161833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.148062    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" event={"ID":"4025821f-f08b-41af-9cd0-45a8b54f5d78","Type":"ContainerDied","Data":"0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75"}
Oct 29 23:02:30.161833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.148112    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8" event={"ID":"4025821f-f08b-41af-9cd0-45a8b54f5d78","Type":"ContainerDied","Data":"d03bb442b265fffbfff4e6605bca80aded046cc97175eafdd9fd5a27c0e9833f"}
Oct 29 23:02:30.161833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.148140    4827 scope.go:122] "RemoveContainer" containerID="0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75"
Oct 29 23:02:30.161833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.148305    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"
Oct 29 23:02:30.622791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.615238    4827 scope.go:122] "RemoveContainer" containerID="0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75"
Oct 29 23:02:30.627710 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:30.623410    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75\": container with ID starting with 0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75 not found: ID does not exist" containerID="0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75"
Oct 29 23:02:30.627710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.623449    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75"} err="failed to get container status \"0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75\": rpc error: code = NotFound desc = could not find container \"0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75\": container with ID starting with 0ad1edf96c924af3c240361d964266f6b2ea37ade878dc3631acd7e3b6cccf75 not found: ID does not exist"
Oct 29 23:02:30.687801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.684966    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"]
Oct 29 23:02:30.742793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.688628    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-jklk8"]
Oct 29 23:02:30.773330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.765471    4827 patch_prober.go:29] interesting pod/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:02:30.773330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:30.765545    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:02:31.110290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.109437    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:31.149644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.149602    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:31.178847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.178268    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:02:31.274867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.274063    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:31.452910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.452863    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:31.677868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.677817    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:31.888891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.834616    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"]
Oct 29 23:02:31.888891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.842118    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy" containerID="cri-o://a85607ed531bd3606b59a547df30c08275c1197a533d2b082f08f93edf63631d" gracePeriod=30
Oct 29 23:02:31.888891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:31.862094    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:32.152962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.152866    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:32.153690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.153664    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:02:32.179887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.179846    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:02:32.180602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.180582    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:02:32.304789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.291071    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:02:32.304789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.291646    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:02:32.506918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.506867    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": context deadline exceeded"
Oct 29 23:02:32.926141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.507159    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:02:32.926141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.697340    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:32.926141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:32.916907    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:33.164357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.164313    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:33.218300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.168335    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:33.232857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.169468    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 29 23:02:33.233050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.180425    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:02:33.253704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.197682    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" path="/var/lib/kubelet/pods/4025821f-f08b-41af-9cd0-45a8b54f5d78/volumes"
Oct 29 23:02:33.254349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.254322    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:02:33.254493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.254405    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:02:33.259262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.259114    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:02:33.307306 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.297894    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:33.313617 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:33.313565    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:02:33.336704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.328129    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:02:33.336704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.328186    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:02:33.350142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.350114    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:02:33.358820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.351635    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:02:33.661040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.659585    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 23:02:33.807452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807422    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-envoy\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807502    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-token\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-token\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807593    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-credential-socket\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807618    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-data\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807675    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-socket\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807712    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-certs\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807804    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istiod-ca-cert\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807844    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-podinfo\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.807911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.807876    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-kube-api-access-btvt6\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-kube-api-access-btvt6\") pod \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\" (UID: \"32fadd7a-083a-4b36-8d31-6d89fd4e9739\") "
Oct 29 23:02:33.808951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.808922    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-data" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:33.809913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.809760    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-socket" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:33.809913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.809896    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istiod-ca-cert" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:33.810202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.810183    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-credential-socket" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:33.810393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.810380    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-certs" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:33.812357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.812325    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-podinfo" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:33.812533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.812506    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-envoy" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:33.812674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.812610    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-token" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:33.813228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.813206    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-kube-api-access-btvt6" pod "32fadd7a-083a-4b36-8d31-6d89fd4e9739" (UID: "32fadd7a-083a-4b36-8d31-6d89fd4e9739"). InnerVolumeSpecName "kube-api-access-btvt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:33.910113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910081    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910109    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910119    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910129    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910137    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910150    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fadd7a-083a-4b36-8d31-6d89fd4e9739-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910163    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910174    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fadd7a-083a-4b36-8d31-6d89fd4e9739-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:33.910372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:33.910188    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btvt6\" (UniqueName: \"kubernetes.io/projected/32fadd7a-083a-4b36-8d31-6d89fd4e9739-kube-api-access-btvt6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.070391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.060913    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"]
Oct 29 23:02:34.089477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.089435    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbe08053-gnw64"]
Oct 29 23:02:34.128451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.128416    4827 generic.go:356] "Generic (PLEG): container finished" podID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerID="385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad" exitCode=0
Oct 29 23:02:34.142575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.130140    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"
Oct 29 23:02:34.143438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.132833    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" event={"ID":"32fadd7a-083a-4b36-8d31-6d89fd4e9739","Type":"ContainerDied","Data":"385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad"}
Oct 29 23:02:34.143685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.143661    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" event={"ID":"32fadd7a-083a-4b36-8d31-6d89fd4e9739","Type":"ContainerDied","Data":"e15aeb02ebeb8c1f82072aed67c18c7166d71ea01bde272a69c40a86081defd6"}
Oct 29 23:02:34.143885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.143832    4827 scope.go:122] "RemoveContainer" containerID="385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad"
Oct 29 23:02:34.257275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.257251    4827 scope.go:122] "RemoveContainer" containerID="978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb"
Oct 29 23:02:34.368786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.353746    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"]
Oct 29 23:02:34.470829 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.467933    4827 scope.go:122] "RemoveContainer" containerID="385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad"
Oct 29 23:02:34.648961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.644552    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:02:34.648961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.644620    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:34.720670 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:34.695490    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad\": container with ID starting with 385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad not found: ID does not exist" containerID="385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad"
Oct 29 23:02:34.720670 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.695536    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad"} err="failed to get container status \"385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad\": rpc error: code = NotFound desc = could not find container \"385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad\": container with ID starting with 385a43adf8a5a6725e4123766dd213e43ac3a7fcaa5c177ef025f372b95522ad not found: ID does not exist"
Oct 29 23:02:34.720670 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.695565    4827 scope.go:122] "RemoveContainer" containerID="978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb"
Oct 29 23:02:34.752701 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:34.751247    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb\": container with ID starting with 978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb not found: ID does not exist" containerID="978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb"
Oct 29 23:02:34.752701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.751299    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb"} err="failed to get container status \"978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb\": rpc error: code = NotFound desc = could not find container \"978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb\": container with ID starting with 978f0e5402246e4f75085351444718f7b9c4dca36b22f4560d9717ad8b2c38cb not found: ID does not exist"
Oct 29 23:02:34.758128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.758097    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6af75e-c9de-4854-9b4f-89370453d05e" path="/var/lib/kubelet/pods/9f6af75e-c9de-4854-9b4f-89370453d05e/volumes"
Oct 29 23:02:34.758557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:34.758536    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3e268-openshift-default-5d495ddfb5-t96gd"]
Oct 29 23:02:35.623728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.623696    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 23:02:35.687791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.673077    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"]
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697068    4827 cpu_manager.go:414] "containerMap: removing container" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697106    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697144    4827 cpu_manager.go:414] "containerMap: removing container" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697152    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697198    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697206    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697217    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697224    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697261    4827 cpu_manager.go:414] "containerMap: removing container" podUID="1a435022-535c-480e-b06b-d6da1508eb97" containerName="maas-api"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697268    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a435022-535c-480e-b06b-d6da1508eb97" containerName="maas-api"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697284    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697298    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697315    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697321    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697594    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy"
Oct 29 23:02:35.702937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.697604    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy"
Oct 29 23:02:35.705223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705196    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy"
Oct 29 23:02:35.705390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705376    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4025821f-f08b-41af-9cd0-45a8b54f5d78" containerName="istio-proxy"
Oct 29 23:02:35.705489 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705479    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy"
Oct 29 23:02:35.705589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705579    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy"
Oct 29 23:02:35.705661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705652    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" containerName="istio-proxy"
Oct 29 23:02:35.705755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705743    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy"
Oct 29 23:02:35.705857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705834    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79b06ae-bc79-45a0-acec-9462a314fba1" containerName="istio-proxy"
Oct 29 23:02:35.705965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.705954    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a435022-535c-480e-b06b-d6da1508eb97" containerName="maas-api"
Oct 29 23:02:35.706045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.706036    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy"
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.721929    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-data\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.721992    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-envoy\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.722077    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-token\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.722175    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-kube-api-access-nlcrt\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-kube-api-access-nlcrt\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.722224    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-socket\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.722301    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-certs\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.722469    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb6d96c-910a-416e-aa2b-87bbf273de36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6d96c-910a-416e-aa2b-87bbf273de36-istiod-ca-cert\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.722621    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-podinfo\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.723115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.722682    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-credential-socket\") pod \"0bb6d96c-910a-416e-aa2b-87bbf273de36\" (UID: \"0bb6d96c-910a-416e-aa2b-87bbf273de36\") "
Oct 29 23:02:35.725903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.724449    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-certs" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.725903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.725028    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-socket" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.725903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.725292    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.726182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.726113    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-kube-api-access-nlcrt" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "kube-api-access-nlcrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:35.728860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.727173    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.728974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.728866    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb6d96c-910a-416e-aa2b-87bbf273de36-istiod-ca-cert" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:35.729451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.729105    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-credential-socket" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.729451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.729316    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-data" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.739856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.739813    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-envoy" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.740018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.739996    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-podinfo" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:35.741942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.741911    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-token" pod "0bb6d96c-910a-416e-aa2b-87bbf273de36" (UID: "0bb6d96c-910a-416e-aa2b-87bbf273de36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:35.742426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.742406    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c88955-openshift-default-dockercfg-49522\""
Oct 29 23:02:35.743948 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.743930    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"]
Oct 29 23:02:35.826791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.826663    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.826791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.826722    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.826986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.826935    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-data\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.827024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.826992    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-token\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.827084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827064    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-envoy\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.827193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827174    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-certs\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.827315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827206    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-credential-socket\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.827315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827279    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-socket\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.827392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827346    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5cq\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-kube-api-access-kz5cq\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.827494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827474    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6d96c-910a-416e-aa2b-87bbf273de36-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.827531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827497    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.827531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827507    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.827531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827520    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.827531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827530    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.827657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827539    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.827657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827547    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlcrt\" (UniqueName: \"kubernetes.io/projected/0bb6d96c-910a-416e-aa2b-87bbf273de36-kube-api-access-nlcrt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.827657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.827555    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6d96c-910a-416e-aa2b-87bbf273de36-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.952795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.950404    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-data\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.967259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.967223    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-data\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976259    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-token\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976348    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-envoy\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976474    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-certs\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976514    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-credential-socket\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976604    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-socket\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976663    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5cq\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-kube-api-access-kz5cq\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976749    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.976820    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.977693    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.980059    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-socket\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.980298    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-certs\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:35.982839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:35.980546    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-credential-socket\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:36.003230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.003200    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:36.037550 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.037522    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"]
Oct 29 23:02:36.072889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.070447    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy"
Oct 29 23:02:36.072889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.070474    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy"
Oct 29 23:02:36.072889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.070492    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy"
Oct 29 23:02:36.072889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.070500    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b985efd-ab05-4960-ae31-13c61bf74b1b" containerName="istio-proxy"
Oct 29 23:02:36.072889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.070743    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerName="istio-proxy"
Oct 29 23:02:36.108830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.088389    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.177078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.177050    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-848fb8-serving-cert\""
Oct 29 23:02:36.187242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.187210    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"]
Oct 29 23:02:36.201953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.191793    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c0a0d-739e-48db-80cd-739869e11aae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.201953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.191875    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.201953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.192070    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-metrics-certs\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.201953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.192177    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfzv\" (UniqueName: \"kubernetes.io/projected/bd9c0a0d-739e-48db-80cd-739869e11aae-kube-api-access-pkfzv\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.270243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.270212    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-token\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.296319    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c0a0d-739e-48db-80cd-739869e11aae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.296402    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.296578    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-metrics-certs\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.296676    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfzv\" (UniqueName: \"kubernetes.io/projected/bd9c0a0d-739e-48db-80cd-739869e11aae-kube-api-access-pkfzv\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.297983    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c0a0d-739e-48db-80cd-739869e11aae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:36.298092    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-serving-cert: secret "maas-api-e2e-ipp-b-848fb8-serving-cert" not found
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:36.298147    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls podName:bd9c0a0d-739e-48db-80cd-739869e11aae nodeName:}" failed. No retries permitted until 2026-10-29 23:02:36.798128588 +0000 UTC m=+1192.421039600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls") pod "maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2" (UID: "bd9c0a0d-739e-48db-80cd-739869e11aae") : secret "maas-api-e2e-ipp-b-848fb8-serving-cert" not found
Oct 29 23:02:36.304842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.300082    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5cq\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-kube-api-access-kz5cq\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:36.307439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.307405    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-envoy\") pod \"e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") " pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:36.312243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.311218    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-metrics-certs\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.321290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.321252    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfzv\" (UniqueName: \"kubernetes.io/projected/bd9c0a0d-739e-48db-80cd-739869e11aae-kube-api-access-pkfzv\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.362473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.362396    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:36.402204 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.402173    4827 generic.go:356] "Generic (PLEG): container finished" podID="0bb6d96c-910a-416e-aa2b-87bbf273de36" containerID="688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57" exitCode=0
Oct 29 23:02:36.402308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.402287    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" event={"ID":"0bb6d96c-910a-416e-aa2b-87bbf273de36","Type":"ContainerDied","Data":"688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57"}
Oct 29 23:02:36.402376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.402336    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp" event={"ID":"0bb6d96c-910a-416e-aa2b-87bbf273de36","Type":"ContainerDied","Data":"d03f89c3a8b0971a6400b6bcdd6fdbd58bcfc056afed38d24db4b1a752257c3c"}
Oct 29 23:02:36.402376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.402366    4827 scope.go:122] "RemoveContainer" containerID="688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57"
Oct 29 23:02:36.402573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.402553    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"
Oct 29 23:02:36.516869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.506965    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"]
Oct 29 23:02:36.617186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.617097    4827 scope.go:122] "RemoveContainer" containerID="2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271"
Oct 29 23:02:36.625777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.625078    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fadd7a-083a-4b36-8d31-6d89fd4e9739" path="/var/lib/kubelet/pods/32fadd7a-083a-4b36-8d31-6d89fd4e9739/volumes"
Oct 29 23:02:36.625777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.625593    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-49dbb5-openshift-default-58d5d56457-wgzmp"]
Oct 29 23:02:36.633788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.633407    4827 scope.go:122] "RemoveContainer" containerID="688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57"
Oct 29 23:02:36.633908 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:36.633859    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57\": container with ID starting with 688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57 not found: ID does not exist" containerID="688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57"
Oct 29 23:02:36.633908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.633893    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57"} err="failed to get container status \"688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57\": rpc error: code = NotFound desc = could not find container \"688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57\": container with ID starting with 688bd6751feb85b457224357177d84257459842bf5784f005ead52d188014f57 not found: ID does not exist"
Oct 29 23:02:36.634037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.633916    4827 scope.go:122] "RemoveContainer" containerID="2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271"
Oct 29 23:02:36.634257 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:36.634234    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271\": container with ID starting with 2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271 not found: ID does not exist" containerID="2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271"
Oct 29 23:02:36.634363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.634265    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271"} err="failed to get container status \"2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271\": rpc error: code = NotFound desc = could not find container \"2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271\": container with ID starting with 2b8dc7908043556900594d34271974248d04e9da0f237b11a715b97b7f201271 not found: ID does not exist"
Oct 29 23:02:36.823339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.823307    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.825648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.825627    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:36.872999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.872974    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"]
Oct 29 23:02:36.873599 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:02:36.873572    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ba4855_3fc6_48d4_81da_448a9ff517af.slice/crio-1dc71b329c18a15677456df3ec757fc665e234e6817c1be059cf9ab1508fc6da WatchSource:0}: Error finding container 1dc71b329c18a15677456df3ec757fc665e234e6817c1be059cf9ab1508fc6da: Status 404 returned error can't find the container with id 1dc71b329c18a15677456df3ec757fc665e234e6817c1be059cf9ab1508fc6da
Oct 29 23:02:36.876639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.876598    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:36.876741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.876689    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:36.876741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.876732    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:36.913846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:36.913824    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:37.237399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:37.237363    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"]
Oct 29 23:02:37.239235 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:02:37.239209    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9c0a0d_739e_48db_80cd_739869e11aae.slice/crio-51860718bb6e7918344d5808098d143d2d774090c33b42d3bab7153349d844f2 WatchSource:0}: Error finding container 51860718bb6e7918344d5808098d143d2d774090c33b42d3bab7153349d844f2: Status 404 returned error can't find the container with id 51860718bb6e7918344d5808098d143d2d774090c33b42d3bab7153349d844f2
Oct 29 23:02:37.417547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:37.417455    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" event={"ID":"c0ba4855-3fc6-48d4-81da-448a9ff517af","Type":"ContainerStarted","Data":"7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da"}
Oct 29 23:02:37.417547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:37.417543    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" event={"ID":"c0ba4855-3fc6-48d4-81da-448a9ff517af","Type":"ContainerStarted","Data":"1dc71b329c18a15677456df3ec757fc665e234e6817c1be059cf9ab1508fc6da"}
Oct 29 23:02:37.424961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:37.424927    4827 generic.go:356] "Generic (PLEG): container finished" podID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerID="a85607ed531bd3606b59a547df30c08275c1197a533d2b082f08f93edf63631d" exitCode=0
Oct 29 23:02:37.425430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:37.425401    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" event={"ID":"469b8d53-8386-4f8b-942b-d0fedb13eb95","Type":"ContainerDied","Data":"a85607ed531bd3606b59a547df30c08275c1197a533d2b082f08f93edf63631d"}
Oct 29 23:02:37.425615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:37.425598    4827 scope.go:122] "RemoveContainer" containerID="889125e7031170fddd986093a37a3dbac36ed904375dd0fe4805ae8545935eb1"
Oct 29 23:02:37.433028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:37.432839    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2" event={"ID":"bd9c0a0d-739e-48db-80cd-739869e11aae","Type":"ContainerStarted","Data":"51860718bb6e7918344d5808098d143d2d774090c33b42d3bab7153349d844f2"}
Oct 29 23:02:38.106795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.105557    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podStartSLOduration=3.105535666 podStartE2EDuration="3.105535666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:37.623961797 +0000 UTC m=+1193.246872817" watchObservedRunningTime="2026-10-29 23:02:38.105535666 +0000 UTC m=+1193.728446687"
Oct 29 23:02:38.109907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.109843    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"]
Oct 29 23:02:38.132786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.115421    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d0b93-g5wp7"]
Oct 29 23:02:38.362982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.362956    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:38.369756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.369735    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:38.387008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.386985    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 23:02:38.409531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.409501    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-certs\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.409687 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.409537    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-socket\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.409687 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.409661    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469b8d53-8386-4f8b-942b-d0fedb13eb95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469b8d53-8386-4f8b-942b-d0fedb13eb95-istiod-ca-cert\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.409866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.409719    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-podinfo\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.409866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.409826    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-certs" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:38.409985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.409953    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-socket" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:38.410039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.409994    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-data\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.410153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.410130    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-kube-api-access-lztng\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-kube-api-access-lztng\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.410226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.410182    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-data" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:38.410226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.410215    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-envoy\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.410576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.410287    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-token\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-token\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.410670 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.410606    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-credential-socket\") pod \"469b8d53-8386-4f8b-942b-d0fedb13eb95\" (UID: \"469b8d53-8386-4f8b-942b-d0fedb13eb95\") "
Oct 29 23:02:38.411130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.411104    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-credential-socket" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:38.411632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.411604    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.411736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.411636    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.411736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.411651    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.411736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.411664    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.411937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.411840    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469b8d53-8386-4f8b-942b-d0fedb13eb95-istiod-ca-cert" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:38.412424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.412401    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-podinfo" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:38.414568 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.414433    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-kube-api-access-lztng" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "kube-api-access-lztng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:38.421213 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.421185    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-token" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:38.582283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.454933    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-envoy" pod "469b8d53-8386-4f8b-942b-d0fedb13eb95" (UID: "469b8d53-8386-4f8b-942b-d0fedb13eb95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:38.582283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.513232    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469b8d53-8386-4f8b-942b-d0fedb13eb95-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.582283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.513259    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.582283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.513328    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lztng\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-kube-api-access-lztng\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.582283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.513344    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.582283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.513370    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469b8d53-8386-4f8b-942b-d0fedb13eb95-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:38.692739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.691503    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"
Oct 29 23:02:38.820714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.820681    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb6d96c-910a-416e-aa2b-87bbf273de36" path="/var/lib/kubelet/pods/0bb6d96c-910a-416e-aa2b-87bbf273de36/volumes"
Oct 29 23:02:38.821395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.821370    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfd6131-e8ec-46af-88e2-c20e52e21c1b" path="/var/lib/kubelet/pods/acfd6131-e8ec-46af-88e2-c20e52e21c1b/volumes"
Oct 29 23:02:38.824953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.822000    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2" podStartSLOduration=3.019095706 podStartE2EDuration="3.821982966s" totalImagesPullingTime="802.88726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:38.729583188 +0000 UTC m=+1194.352494213" watchObservedRunningTime="2026-10-29 23:02:38.821982966 +0000 UTC m=+1194.444893987"
Oct 29 23:02:38.826526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.826500    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2" event={"ID":"bd9c0a0d-739e-48db-80cd-739869e11aae","Type":"ContainerStarted","Data":"c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75"}
Oct 29 23:02:38.826700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.826680    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b" event={"ID":"469b8d53-8386-4f8b-942b-d0fedb13eb95","Type":"ContainerDied","Data":"139a0f02b5832499821b8d85df967a1ea3f6a412b34f58f2ec49a6a327c9e595"}
Oct 29 23:02:38.826844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.826832    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:38.826949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.826938    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:38.827100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.827088    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:02:38.827246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.827234    4827 scope.go:122] "RemoveContainer" containerID="a85607ed531bd3606b59a547df30c08275c1197a533d2b082f08f93edf63631d"
Oct 29 23:02:38.862563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.862535    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"]
Oct 29 23:02:38.876808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:38.869284    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-f9n4b"]
Oct 29 23:02:40.679805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:40.679735    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" path="/var/lib/kubelet/pods/469b8d53-8386-4f8b-942b-d0fedb13eb95/volumes"
Oct 29 23:02:41.562876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.562835    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"]
Oct 29 23:02:41.564165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.564133    4827 cpu_manager.go:414] "containerMap: removing container" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy"
Oct 29 23:02:41.564334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.564319    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy"
Oct 29 23:02:41.564466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.564455    4827 cpu_manager.go:414] "containerMap: removing container" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy"
Oct 29 23:02:41.564559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.564544    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy"
Oct 29 23:02:41.564886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.564870    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy"
Oct 29 23:02:41.565019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.565008    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="469b8d53-8386-4f8b-942b-d0fedb13eb95" containerName="istio-proxy"
Oct 29 23:02:41.573806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.573785    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.704900    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705029    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705227    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705257    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705292    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705318    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705346    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e856e7-5135-4814-beaf-7f4c10708fd3-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705437    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tztnh\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-kube-api-access-tztnh\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.711953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.705538    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.766815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.760890    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"]
Oct 29 23:02:41.875840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.865046    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.878291    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.878515    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.878559    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.878588    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e856e7-5135-4814-beaf-7f4c10708fd3-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.878837    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tztnh\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-kube-api-access-tztnh\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.879015    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.879091    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.879245    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.879888    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.880366    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.883891    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e856e7-5135-4814-beaf-7f4c10708fd3-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.884168    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.885798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.884397    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.922185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.920739    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:41.992851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:41.992815    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"]
Oct 29 23:02:42.065835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:42.061326    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:42.065835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:42.065482    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:42.068008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:42.067887    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tztnh\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-kube-api-access-tztnh\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:42.224816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:42.224695    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:43.006593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.006557    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"]
Oct 29 23:02:43.007100 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:02:43.006553    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e856e7_5135_4814_beaf_7f4c10708fd3.slice/crio-d3d219c799b590ec2554a6eed3e95696bf50345796f02ffbe8d8bf8378ea4f8b WatchSource:0}: Error finding container d3d219c799b590ec2554a6eed3e95696bf50345796f02ffbe8d8bf8378ea4f8b: Status 404 returned error can't find the container with id d3d219c799b590ec2554a6eed3e95696bf50345796f02ffbe8d8bf8378ea4f8b
Oct 29 23:02:43.010887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.010849    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:43.010988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.010930    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:43.010988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.010976    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:43.440139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.438313    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": read tcp 10.134.0.2:32858->10.134.0.159:15021: read: connection reset by peer" start-of-body=
Oct 29 23:02:43.440139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.438369    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": read tcp 10.134.0.2:32858->10.134.0.159:15021: read: connection reset by peer"
Oct 29 23:02:43.792557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.792464    4827 generic.go:356] "Generic (PLEG): container finished" podID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerID="89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48" exitCode=0
Oct 29 23:02:43.792862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.792549    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" event={"ID":"bb46a1dd-0383-4d4b-840e-6e45e6c76046","Type":"ContainerDied","Data":"89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48"}
Oct 29 23:02:43.793151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.793130    4827 scope.go:122] "RemoveContainer" containerID="89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48"
Oct 29 23:02:43.794627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.794606    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" event={"ID":"07e856e7-5135-4814-beaf-7f4c10708fd3","Type":"ContainerStarted","Data":"7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201"}
Oct 29 23:02:43.794720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.794655    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" event={"ID":"07e856e7-5135-4814-beaf-7f4c10708fd3","Type":"ContainerStarted","Data":"d3d219c799b590ec2554a6eed3e95696bf50345796f02ffbe8d8bf8378ea4f8b"}
Oct 29 23:02:43.795261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.795236    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:43.795329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.795304    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:43.795368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:43.795341    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:02:44.040839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.017908    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podStartSLOduration=3.017886072 podStartE2EDuration="3.017886072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:43.924059608 +0000 UTC m=+1199.546970643" watchObservedRunningTime="2026-10-29 23:02:44.017886072 +0000 UTC m=+1199.640797093"
Oct 29 23:02:44.225300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.225268    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:44.226666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.226637    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 23:02:44.226935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.226908    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 23:02:44.729908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.729868    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:02:44.893790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.864348    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" event={"ID":"bb46a1dd-0383-4d4b-840e-6e45e6c76046","Type":"ContainerStarted","Data":"4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe"}
Oct 29 23:02:44.926211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.926183    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:02:44.926757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:44.926739    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:02:45.243481 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:45.243449    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 23:02:45.244162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:45.244111    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 23:02:45.430703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:45.430669    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:02:45.448868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:45.448831    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 29 23:02:45.462258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:45.462213    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 29 23:02:46.226058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.226027    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 23:02:46.226244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.226084    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 23:02:46.365617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.365577    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"]
Oct 29 23:02:46.366124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.365868    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy" containerID="cri-o://ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515" gracePeriod=30
Oct 29 23:02:46.458653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.458592    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 29 23:02:46.458921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.458658    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 29 23:02:46.741792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.737919    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"]
Oct 29 23:02:46.741792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.738178    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy" containerID="cri-o://44c04dd6445d22cda20fd0799a10a8e5ff322d323f014a781351bd0b70f5044d" gracePeriod=30
Oct 29 23:02:46.789803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.784814    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"]
Oct 29 23:02:46.789803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:46.785061    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy" containerID="cri-o://cd16e51eab223f756d9dca702be3ca3424a853b8f6e6424b5426af37b119b4ef" gracePeriod=30
Oct 29 23:02:47.001951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.001843    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq" podUID="25899eb6-4369-419b-8485-ce80c082afd0" containerName="payload-pre-processing" containerID="cri-o://4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78" gracePeriod=25
Oct 29 23:02:47.221517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.221485    4827 scope.go:122] "RemoveContainer" containerID="b569fde66eef328e1206ddd76336a56cd981ff8b874b9d5fabff220dcd3b4b69"
Oct 29 23:02:47.226118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.226092    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 23:02:47.226228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.226157    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 23:02:47.233724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.233710    4827 scope.go:122] "RemoveContainer" containerID="7e77b4b138d7fd3466adecd7d4b78c5c4a87858dcf83f8ae255e057354ab732d"
Oct 29 23:02:47.256449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.255537    4827 scope.go:122] "RemoveContainer" containerID="d882bf8b088022a8cd04e8b03614782454b7310c65d3e699f7a5c4426be75f87"
Oct 29 23:02:47.334091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.331114    4827 scope.go:122] "RemoveContainer" containerID="42a7525d89b11e618378a425fa4bbd82d675bb06c9afb8e85060843c846921d8"
Oct 29 23:02:47.348531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.348508    4827 scope.go:122] "RemoveContainer" containerID="2fbc1d7c29da9f8b71d170c2cb6d746096481d148b825e6804efbb6118daa527"
Oct 29 23:02:47.426214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.425744    4827 scope.go:122] "RemoveContainer" containerID="4a7d5f67c4fe71a7a118e090a6c0c5e0c5c7196ac5e58f57ae6c7e9e7ae91d62"
Oct 29 23:02:47.429254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.428720    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 29 23:02:47.429254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.428791    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 29 23:02:47.493802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.490975    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:02:47.627629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.627602    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25899eb6-4369-419b-8485-ce80c082afd0-kube-api-access-ggd8m\" (UniqueName: \"kubernetes.io/projected/25899eb6-4369-419b-8485-ce80c082afd0-kube-api-access-ggd8m\") pod \"25899eb6-4369-419b-8485-ce80c082afd0\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") "
Oct 29 23:02:47.627851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.627829    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25899eb6-4369-419b-8485-ce80c082afd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25899eb6-4369-419b-8485-ce80c082afd0-plugins-config-volume\") pod \"25899eb6-4369-419b-8485-ce80c082afd0\" (UID: \"25899eb6-4369-419b-8485-ce80c082afd0\") "
Oct 29 23:02:47.628338 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.628208    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25899eb6-4369-419b-8485-ce80c082afd0-plugins-config-volume" pod "25899eb6-4369-419b-8485-ce80c082afd0" (UID: "25899eb6-4369-419b-8485-ce80c082afd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:47.628669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.628645    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25899eb6-4369-419b-8485-ce80c082afd0-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:47.631264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.631242    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25899eb6-4369-419b-8485-ce80c082afd0-kube-api-access-ggd8m" pod "25899eb6-4369-419b-8485-ce80c082afd0" (UID: "25899eb6-4369-419b-8485-ce80c082afd0"). InnerVolumeSpecName "kube-api-access-ggd8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:47.730549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.730525    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggd8m\" (UniqueName: \"kubernetes.io/projected/25899eb6-4369-419b-8485-ce80c082afd0-kube-api-access-ggd8m\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:47.938733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.938700    4827 generic.go:356] "Generic (PLEG): container finished" podID="25899eb6-4369-419b-8485-ce80c082afd0" containerID="4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78" exitCode=0
Oct 29 23:02:47.938983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.938955    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq" event={"ID":"25899eb6-4369-419b-8485-ce80c082afd0","Type":"ContainerDied","Data":"4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78"}
Oct 29 23:02:47.939075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.939025    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq" event={"ID":"25899eb6-4369-419b-8485-ce80c082afd0","Type":"ContainerDied","Data":"bb36a8fa455864d8a84e9d631b11ef32c768ad6c082bba0fcf577aaf19b856cd"}
Oct 29 23:02:47.939135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.939077    4827 scope.go:122] "RemoveContainer" containerID="4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78"
Oct 29 23:02:47.939193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.939141    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"
Oct 29 23:02:47.950827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.950807    4827 scope.go:122] "RemoveContainer" containerID="4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78"
Oct 29 23:02:47.951109 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:47.951084    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78\": container with ID starting with 4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78 not found: ID does not exist" containerID="4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78"
Oct 29 23:02:47.951210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:47.951116    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78"} err="failed to get container status \"4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78\": rpc error: code = NotFound desc = could not find container \"4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78\": container with ID starting with 4bd7c564c5be32a203f8d95a5a3254835c453e21ed568012412c48223b6d0a78 not found: ID does not exist"
Oct 29 23:02:48.012797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:48.011814    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"]
Oct 29 23:02:48.019274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:48.019247    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-774bff-78878f7875-xbflq"]
Oct 29 23:02:48.225362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:48.225274    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 23:02:48.225362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:48.225338    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 23:02:48.429203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:48.429167    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 29 23:02:48.430939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:48.430905    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 29 23:02:48.538484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:48.535987    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25899eb6-4369-419b-8485-ce80c082afd0" path="/var/lib/kubelet/pods/25899eb6-4369-419b-8485-ce80c082afd0/volumes"
Oct 29 23:02:49.225336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:49.225305    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 23:02:49.225622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:49.225601    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 23:02:49.429390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:49.429352    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 29 23:02:49.429960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:49.429400    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 29 23:02:50.231106 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:50.231071    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:50.231646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:50.231624    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:50.232494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:50.232475    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:02:50.429339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:50.429307    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 29 23:02:50.429503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:50.429382    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 29 23:02:51.433744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.433712    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:02:51.434271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.434252    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:02:51.435046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.435029    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:02:51.753637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.753601    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 23:02:51.847405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.847374    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-token\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-token\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.847588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.847475    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-envoy\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.847588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.847545    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-kube-api-access-hmtm8\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-kube-api-access-hmtm8\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.847819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.847802    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-socket\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.847902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.847844    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-certs\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.847980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.847966    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-credential-socket\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.848068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.848051    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-data\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.848128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.848112    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4fb03b-5491-4d64-a680-2ab271a593b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4fb03b-5491-4d64-a680-2ab271a593b5-istiod-ca-cert\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.848187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.848142    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-podinfo\") pod \"7e4fb03b-5491-4d64-a680-2ab271a593b5\" (UID: \"7e4fb03b-5491-4d64-a680-2ab271a593b5\") "
Oct 29 23:02:51.848351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.848176    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-socket" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:51.848402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.848371    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-credential-socket" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:51.849034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.848974    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.849034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.849008    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.849034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.848972    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-data" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:51.849034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.849008    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4fb03b-5491-4d64-a680-2ab271a593b5-istiod-ca-cert" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:51.849411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.849158    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-certs" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:51.850371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.850314    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-token" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:51.851721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.851663    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-kube-api-access-hmtm8" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "kube-api-access-hmtm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:51.851864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.851757    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-podinfo" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:51.852592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.852490    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-envoy" pod "7e4fb03b-5491-4d64-a680-2ab271a593b5" (UID: "7e4fb03b-5491-4d64-a680-2ab271a593b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:51.950341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.950260    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.950341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.950287    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmtm8\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-kube-api-access-hmtm8\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.950341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.950303    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.950341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.950314    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.950341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.950323    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4fb03b-5491-4d64-a680-2ab271a593b5-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.950341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.950331    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:51.950341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:51.950338    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4fb03b-5491-4d64-a680-2ab271a593b5-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.014616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.014374    4827 generic.go:356] "Generic (PLEG): container finished" podID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerID="44c04dd6445d22cda20fd0799a10a8e5ff322d323f014a781351bd0b70f5044d" exitCode=0
Oct 29 23:02:52.014616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.014475    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" event={"ID":"2c5f9825-0f08-40ce-a49d-d8039190f8e3","Type":"ContainerDied","Data":"44c04dd6445d22cda20fd0799a10a8e5ff322d323f014a781351bd0b70f5044d"}
Oct 29 23:02:52.019881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.017087    4827 generic.go:356] "Generic (PLEG): container finished" podID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerID="ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515" exitCode=0
Oct 29 23:02:52.019881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.017167    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" event={"ID":"7e4fb03b-5491-4d64-a680-2ab271a593b5","Type":"ContainerDied","Data":"ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515"}
Oct 29 23:02:52.019881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.017213    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2" event={"ID":"7e4fb03b-5491-4d64-a680-2ab271a593b5","Type":"ContainerDied","Data":"5de91f0f19e9ffdabdcf272a22fa7780a8024546fa3362c42a32a949c7db8560"}
Oct 29 23:02:52.019881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.017246    4827 scope.go:122] "RemoveContainer" containerID="ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515"
Oct 29 23:02:52.019881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.017419    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"
Oct 29 23:02:52.023900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.023224    4827 generic.go:356] "Generic (PLEG): container finished" podID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerID="cd16e51eab223f756d9dca702be3ca3424a853b8f6e6424b5426af37b119b4ef" exitCode=0
Oct 29 23:02:52.025084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.024047    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" event={"ID":"3293ebd3-22cd-4cec-b069-eb54d6687ec9","Type":"ContainerDied","Data":"cd16e51eab223f756d9dca702be3ca3424a853b8f6e6424b5426af37b119b4ef"}
Oct 29 23:02:52.048437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.048408    4827 scope.go:122] "RemoveContainer" containerID="ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515"
Oct 29 23:02:52.049663 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:52.049639    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515\": container with ID starting with ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515 not found: ID does not exist" containerID="ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515"
Oct 29 23:02:52.049790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.049671    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515"} err="failed to get container status \"ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515\": rpc error: code = NotFound desc = could not find container \"ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515\": container with ID starting with ec6c010e5b0e8e056df2730605938e8b0314f53175bb89c3f401e0ef1c1b1515 not found: ID does not exist"
Oct 29 23:02:52.060427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.060404    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 23:02:52.102650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.102566    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"]
Oct 29 23:02:52.108237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.108213    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nchx2"]
Oct 29 23:02:52.115466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.115444    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.155830    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-certs\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.155945    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-envoy\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.155997    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-data\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.156073    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istiod-ca-cert\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.156125    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-token\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-token\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.156165    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-credential-socket\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.156290    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-kube-api-access-thcq6\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-kube-api-access-thcq6\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.156362    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-socket\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.156700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.156402    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-podinfo\") pod \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\" (UID: \"3293ebd3-22cd-4cec-b069-eb54d6687ec9\") "
Oct 29 23:02:52.157396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.156869    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istiod-ca-cert" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:52.157396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.157121    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-certs" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.157712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.157685    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.157885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.157715    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.158339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.158315    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-data" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.159141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.159118    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-socket" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.159275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.159130    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-credential-socket" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.160116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.160083    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-token" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:52.160867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.160845    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-envoy" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.161906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.161882    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-podinfo" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:52.173050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.173024    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-kube-api-access-thcq6" pod "3293ebd3-22cd-4cec-b069-eb54d6687ec9" (UID: "3293ebd3-22cd-4cec-b069-eb54d6687ec9"). InnerVolumeSpecName "kube-api-access-thcq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:52.258502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258411    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-socket\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.258502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258502    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-envoy\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.258733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258523    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-certs\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.258733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258585    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-podinfo\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.258733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258609    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-token\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.258733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258622    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-socket" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.258733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258659    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-kube-api-access-s8469\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-kube-api-access-s8469\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.258733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258721    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istiod-ca-cert\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.259077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258875    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-credential-socket\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.259077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258914    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-data\") pod \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\" (UID: \"2c5f9825-0f08-40ce-a49d-d8039190f8e3\") "
Oct 29 23:02:52.259077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.258955    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-certs" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.259233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259190    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istiod-ca-cert" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:52.259383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259363    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-credential-socket" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.259455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259437    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-data" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.259681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259662    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.259784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259688    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thcq6\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-kube-api-access-thcq6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.259784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259702    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.259784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259715    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.259784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259728    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.259784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259740    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.259784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259753    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.260088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259907    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.260088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259928    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.260088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259942    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3293ebd3-22cd-4cec-b069-eb54d6687ec9-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.260088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259954    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.260088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.259967    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3293ebd3-22cd-4cec-b069-eb54d6687ec9-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.261188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.261164    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-envoy" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:52.261289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.261274    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-token" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:52.261369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.261332    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-podinfo" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:52.261369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.261347    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-kube-api-access-s8469" pod "2c5f9825-0f08-40ce-a49d-d8039190f8e3" (UID: "2c5f9825-0f08-40ce-a49d-d8039190f8e3"). InnerVolumeSpecName "kube-api-access-s8469". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:52.361345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.361315    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.361671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.361656    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.361757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.361746    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.361860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.361846    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8469\" (UniqueName: \"kubernetes.io/projected/2c5f9825-0f08-40ce-a49d-d8039190f8e3-kube-api-access-s8469\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:02:52.494514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:52.494480    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" path="/var/lib/kubelet/pods/7e4fb03b-5491-4d64-a680-2ab271a593b5/volumes"
Oct 29 23:02:53.035986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.035957    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"
Oct 29 23:02:53.036178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.035955    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv" event={"ID":"2c5f9825-0f08-40ce-a49d-d8039190f8e3","Type":"ContainerDied","Data":"c7b7f67dc9119bf78442543536df362abfd640a1b108a4948cb7c4a5ab8944d2"}
Oct 29 23:02:53.036178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.036102    4827 scope.go:122] "RemoveContainer" containerID="44c04dd6445d22cda20fd0799a10a8e5ff322d323f014a781351bd0b70f5044d"
Oct 29 23:02:53.039058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.039034    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"
Oct 29 23:02:53.039283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.039258    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8" event={"ID":"3293ebd3-22cd-4cec-b069-eb54d6687ec9","Type":"ContainerDied","Data":"d4ed78b858d405e2bd77e8c3ace1380d1778ac530f7aba60c52108fc37ce7fa0"}
Oct 29 23:02:53.047662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.047636    4827 scope.go:122] "RemoveContainer" containerID="cd16e51eab223f756d9dca702be3ca3424a853b8f6e6424b5426af37b119b4ef"
Oct 29 23:02:53.101638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.101612    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"]
Oct 29 23:02:53.105362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.105342    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d75k8"]
Oct 29 23:02:53.141396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.141362    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"]
Oct 29 23:02:53.146172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:53.146141    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-nrvhv"]
Oct 29 23:02:54.494599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:54.494551    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" path="/var/lib/kubelet/pods/2c5f9825-0f08-40ce-a49d-d8039190f8e3/volumes"
Oct 29 23:02:54.495468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:54.495441    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" path="/var/lib/kubelet/pods/3293ebd3-22cd-4cec-b069-eb54d6687ec9/volumes"
Oct 29 23:02:59.023815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.023724    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"]
Oct 29 23:02:59.024483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024465    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy"
Oct 29 23:02:59.024523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024484    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy"
Oct 29 23:02:59.024523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024514    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy"
Oct 29 23:02:59.024523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024520    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy"
Oct 29 23:02:59.024638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024529    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy"
Oct 29 23:02:59.024638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024534    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy"
Oct 29 23:02:59.024638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024557    4827 cpu_manager.go:414] "containerMap: removing container" podUID="25899eb6-4369-419b-8485-ce80c082afd0" containerName="payload-pre-processing"
Oct 29 23:02:59.024638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024562    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25899eb6-4369-419b-8485-ce80c082afd0" containerName="payload-pre-processing"
Oct 29 23:02:59.024781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024649    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="25899eb6-4369-419b-8485-ce80c082afd0" containerName="payload-pre-processing"
Oct 29 23:02:59.024781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024663    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3293ebd3-22cd-4cec-b069-eb54d6687ec9" containerName="istio-proxy"
Oct 29 23:02:59.024781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024679    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5f9825-0f08-40ce-a49d-d8039190f8e3" containerName="istio-proxy"
Oct 29 23:02:59.024781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.024689    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4fb03b-5491-4d64-a680-2ab271a593b5" containerName="istio-proxy"
Oct 29 23:02:59.028206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.028183    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.048469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.048444    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"]
Oct 29 23:02:59.051021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.051001    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-53a768-serving-cert\""
Oct 29 23:02:59.065293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.065260    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.065902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.065330    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-metrics-certs\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.065902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.065413    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e73970-3f74-470e-9620-fd257b432d9f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.065902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.065652    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ksv\" (UniqueName: \"kubernetes.io/projected/35e73970-3f74-470e-9620-fd257b432d9f-kube-api-access-x9ksv\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.166657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.166630    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.166861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.166680    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-metrics-certs\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.166861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.166703    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e73970-3f74-470e-9620-fd257b432d9f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.166861 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:59.166818    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-serving-cert: secret "maas-api-e2e-praxis-a-53a768-serving-cert" not found
Oct 29 23:02:59.166979 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:02:59.166905    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls podName:35e73970-3f74-470e-9620-fd257b432d9f nodeName:}" failed. No retries permitted until 2026-10-29 23:02:59.666885678 +0000 UTC m=+1215.289796684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls") pod "maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd" (UID: "35e73970-3f74-470e-9620-fd257b432d9f") : secret "maas-api-e2e-praxis-a-53a768-serving-cert" not found
Oct 29 23:02:59.167024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.166973    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ksv\" (UniqueName: \"kubernetes.io/projected/35e73970-3f74-470e-9620-fd257b432d9f-kube-api-access-x9ksv\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.167450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.167422    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e73970-3f74-470e-9620-fd257b432d9f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.169128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.169107    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-metrics-certs\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.243813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.234606    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ksv\" (UniqueName: \"kubernetes.io/projected/35e73970-3f74-470e-9620-fd257b432d9f-kube-api-access-x9ksv\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.681052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.681017    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.684208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.684181    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:02:59.941717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:02:59.941640    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:03:00.342347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:00.342313    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"]
Oct 29 23:03:01.273803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:01.273727    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd" event={"ID":"35e73970-3f74-470e-9620-fd257b432d9f","Type":"ContainerStarted","Data":"abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6"}
Oct 29 23:03:01.273980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:01.273867    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd" event={"ID":"35e73970-3f74-470e-9620-fd257b432d9f","Type":"ContainerStarted","Data":"592b7eb628b809b68b0266b18b951a371900b76847324c789961c88ac6e74d5e"}
Oct 29 23:03:01.274078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:01.273989    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:03:01.316809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:01.316727    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd" podStartSLOduration=2.981004562 podStartE2EDuration="3.316712549s" totalImagesPullingTime="335.707987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:01.316351599 +0000 UTC m=+1216.939262620" watchObservedRunningTime="2026-10-29 23:03:01.316712549 +0000 UTC m=+1216.939623572"
Oct 29 23:03:02.829267 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:02.829235    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:03:02.896835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.896800    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"]
Oct 29 23:03:02.901334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.901311    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:02.910169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.909617    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpp45\" (UniqueName: \"kubernetes.io/projected/5f772756-617a-4aef-baae-b56289fd1de2-kube-api-access-zpp45\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:02.910169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.909882    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f772756-617a-4aef-baae-b56289fd1de2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:02.910169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.910102    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:02.910296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.910186    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-metrics-certs\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:02.925030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.925001    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"]
Oct 29 23:03:02.931293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:02.931230    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1e1c46-serving-cert\""
Oct 29 23:03:03.011912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.011874    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.012125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.011977    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-metrics-certs\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.012125 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:03.012028    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-serving-cert: secret "maas-api-e2e-praxis-b-1e1c46-serving-cert" not found
Oct 29 23:03:03.012125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.012076    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpp45\" (UniqueName: \"kubernetes.io/projected/5f772756-617a-4aef-baae-b56289fd1de2-kube-api-access-zpp45\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.012125 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:03.012101    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls podName:5f772756-617a-4aef-baae-b56289fd1de2 nodeName:}" failed. No retries permitted until 2026-10-29 23:03:03.512084751 +0000 UTC m=+1219.134995748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls") pod "maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv" (UID: "5f772756-617a-4aef-baae-b56289fd1de2") : secret "maas-api-e2e-praxis-b-1e1c46-serving-cert" not found
Oct 29 23:03:03.012357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.012293    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f772756-617a-4aef-baae-b56289fd1de2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.013396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.013371    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f772756-617a-4aef-baae-b56289fd1de2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.014447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.014428    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-metrics-certs\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.037714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.037693    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpp45\" (UniqueName: \"kubernetes.io/projected/5f772756-617a-4aef-baae-b56289fd1de2-kube-api-access-zpp45\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.528240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.528206    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.530630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.530608    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:03.816321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:03.816244    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:04.353646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:04.327954    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"]
Oct 29 23:03:04.515055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:04.515018    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv" event={"ID":"5f772756-617a-4aef-baae-b56289fd1de2","Type":"ContainerStarted","Data":"deeff343b6f3785c0d996c44adc10a2349ce11bad89b177526172d07661ff508"}
Oct 29 23:03:05.514894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:05.514859    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv" event={"ID":"5f772756-617a-4aef-baae-b56289fd1de2","Type":"ContainerStarted","Data":"5e8723713fb78ccf62978ee8ebb7644b957bb7bcafed6ff2fc087e2b5a2430d6"}
Oct 29 23:03:05.515269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:05.514931    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:05.560877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:05.560826    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv" podStartSLOduration=3.15416199 podStartE2EDuration="3.560813163s" totalImagesPullingTime="406.651173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:05.556927606 +0000 UTC m=+1221.179838626" watchObservedRunningTime="2026-10-29 23:03:05.560813163 +0000 UTC m=+1221.183724201"
Oct 29 23:03:06.958344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:06.958294    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"]
Oct 29 23:03:07.106925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.094317    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"]
Oct 29 23:03:07.107121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.106261    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.138049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.122208    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d67deb-serving-cert\""
Oct 29 23:03:07.468803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.466948    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9f86\" (UniqueName: \"kubernetes.io/projected/9d375c9d-14dd-4f68-920d-972e801e909f-kube-api-access-j9f86\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.468803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.467031    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-metrics-certs\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.468803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.467196    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.468803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.467495    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d375c9d-14dd-4f68-920d-972e801e909f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.499994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.499958    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:03:07.591023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.580426    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d375c9d-14dd-4f68-920d-972e801e909f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.591023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.580485    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9f86\" (UniqueName: \"kubernetes.io/projected/9d375c9d-14dd-4f68-920d-972e801e909f-kube-api-access-j9f86\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.591023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.580528    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-metrics-certs\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.591023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.580656    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.591023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.583369    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d375c9d-14dd-4f68-920d-972e801e909f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.598459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.598421    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.599456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.599433    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-metrics-certs\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.606473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.606443    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9f86\" (UniqueName: \"kubernetes.io/projected/9d375c9d-14dd-4f68-920d-972e801e909f-kube-api-access-j9f86\") pod \"maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:07.709160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:07.709068    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:08.114629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:08.114582    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"]
Oct 29 23:03:08.115401 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:08.115356    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d375c9d_14dd_4f68_920d_972e801e909f.slice/crio-1e53d2c2427bec986ca8fd2560a9cabc7566bbdf7897f90015c31777b7282a95 WatchSource:0}: Error finding container 1e53d2c2427bec986ca8fd2560a9cabc7566bbdf7897f90015c31777b7282a95: Status 404 returned error can't find the container with id 1e53d2c2427bec986ca8fd2560a9cabc7566bbdf7897f90015c31777b7282a95
Oct 29 23:03:08.583903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:08.583865    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp" event={"ID":"9d375c9d-14dd-4f68-920d-972e801e909f","Type":"ContainerStarted","Data":"1e53d2c2427bec986ca8fd2560a9cabc7566bbdf7897f90015c31777b7282a95"}
Oct 29 23:03:09.179649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.179612    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"]
Oct 29 23:03:09.183253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.183233    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.205286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.205264    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"]
Oct 29 23:03:09.212036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.212003    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c88955-serving-cert\""
Oct 29 23:03:09.275309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.275275    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.275456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.275346    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21085ba6-e60a-4c67-a3ba-32592259b601-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.275556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.275532    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jw6s\" (UniqueName: \"kubernetes.io/projected/21085ba6-e60a-4c67-a3ba-32592259b601-kube-api-access-7jw6s\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.275608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.275593    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-metrics-certs\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.376989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.376957    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.377176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.377018    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21085ba6-e60a-4c67-a3ba-32592259b601-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.377176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.377131    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw6s\" (UniqueName: \"kubernetes.io/projected/21085ba6-e60a-4c67-a3ba-32592259b601-kube-api-access-7jw6s\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.377288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.377173    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-metrics-certs\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.377959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.377931    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21085ba6-e60a-4c67-a3ba-32592259b601-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.380365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.380307    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-metrics-certs\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.380365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.380304    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.403825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.403738    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw6s\" (UniqueName: \"kubernetes.io/projected/21085ba6-e60a-4c67-a3ba-32592259b601-kube-api-access-7jw6s\") pod \"maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.503646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.503318    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:09.606243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.606111    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp" event={"ID":"9d375c9d-14dd-4f68-920d-972e801e909f","Type":"ContainerStarted","Data":"c0409875da5ae2c50f4b3eb44d2ab938ca91cef001f3d61d6e421d7ea074baac"}
Oct 29 23:03:09.606243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.606201    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:09.936508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.936456    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp" podStartSLOduration=3.508905155 podStartE2EDuration="3.936441422s" totalImagesPullingTime="427.536267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:09.693420511 +0000 UTC m=+1225.316331531" watchObservedRunningTime="2026-10-29 23:03:09.936441422 +0000 UTC m=+1225.559352442"
Oct 29 23:03:09.941815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:09.941741    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"]
Oct 29 23:03:09.941934 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:09.941882    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21085ba6_e60a_4c67_a3ba_32592259b601.slice/crio-87b7405618a6572d69d359e6c711be49e8c2794d8eba89321721295de78d3c89 WatchSource:0}: Error finding container 87b7405618a6572d69d359e6c711be49e8c2794d8eba89321721295de78d3c89: Status 404 returned error can't find the container with id 87b7405618a6572d69d359e6c711be49e8c2794d8eba89321721295de78d3c89
Oct 29 23:03:10.624471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:10.624427    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm" event={"ID":"21085ba6-e60a-4c67-a3ba-32592259b601","Type":"ContainerStarted","Data":"5b45bff2b0e81e2b324b96088b98f5324f787c6eda3dec3dc4ac0220869a90c4"}
Oct 29 23:03:10.624998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:10.624485    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm" event={"ID":"21085ba6-e60a-4c67-a3ba-32592259b601","Type":"ContainerStarted","Data":"87b7405618a6572d69d359e6c711be49e8c2794d8eba89321721295de78d3c89"}
Oct 29 23:03:10.624998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:10.624939    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:10.671330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:10.671270    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm" podStartSLOduration=1.319122602 podStartE2EDuration="1.671255282s" totalImagesPullingTime="352.13268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:10.667721279 +0000 UTC m=+1226.290632301" watchObservedRunningTime="2026-10-29 23:03:10.671255282 +0000 UTC m=+1226.294166298"
Oct 29 23:03:11.624789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.620946    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:03:11.667824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.664074    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"]
Oct 29 23:03:11.674412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.674387    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"]
Oct 29 23:03:11.674530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.674511    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774007    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774112    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774170    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2fbac-aceb-4938-9032-537bf3808f96-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774259    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774353    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774511    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbh9t\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-kube-api-access-hbh9t\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774543    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774562    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.774643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.774623    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2fbac-aceb-4938-9032-537bf3808f96-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876174    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876231    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876323    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbh9t\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-kube-api-access-hbh9t\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876364    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876390    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876415    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2fbac-aceb-4938-9032-537bf3808f96-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876566    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876651    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876707    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876733    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2fbac-aceb-4938-9032-537bf3808f96-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876836    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.876987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.876932    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.877047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.877019    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.877417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.877397    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2fbac-aceb-4938-9032-537bf3808f96-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.879286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.879261    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2fbac-aceb-4938-9032-537bf3808f96-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.879401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.879346    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.899325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.899295    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.899467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.899353    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbh9t\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-kube-api-access-hbh9t\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:11.994883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:11.994851    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:12.317662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.317640    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"]
Oct 29 23:03:12.319325 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:12.319298    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c2fbac_aceb_4938_9032_537bf3808f96.slice/crio-50b7bd17722e57b688eb8cece2230f4e4073dab5896279bf8d3b2bc239a606a4 WatchSource:0}: Error finding container 50b7bd17722e57b688eb8cece2230f4e4073dab5896279bf8d3b2bc239a606a4: Status 404 returned error can't find the container with id 50b7bd17722e57b688eb8cece2230f4e4073dab5896279bf8d3b2bc239a606a4
Oct 29 23:03:12.322130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.322094    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:12.322208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.322162    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:12.322208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.322193    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:12.668647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.668610    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" event={"ID":"85c2fbac-aceb-4938-9032-537bf3808f96","Type":"ContainerStarted","Data":"7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650"}
Oct 29 23:03:12.669088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.668664    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" event={"ID":"85c2fbac-aceb-4938-9032-537bf3808f96","Type":"ContainerStarted","Data":"50b7bd17722e57b688eb8cece2230f4e4073dab5896279bf8d3b2bc239a606a4"}
Oct 29 23:03:12.720392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.720344    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" podStartSLOduration=1.7203265540000001 podStartE2EDuration="1.720326554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:12.713849493 +0000 UTC m=+1228.336760512" watchObservedRunningTime="2026-10-29 23:03:12.720326554 +0000 UTC m=+1228.343237585"
Oct 29 23:03:12.995701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:12.995619    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:13.286844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:13.282987    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-tjghh_8160871b-24d7-4b6b-bbd5-64ed33ae6382/payload-processing/0.log"
Oct 29 23:03:13.510653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:13.510620    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-tjghh_8160871b-24d7-4b6b-bbd5-64ed33ae6382/payload-processing/0.log"
Oct 29 23:03:13.997244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:13.997215    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:03:13.997680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:13.997265    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 29 23:03:14.706616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.706586    4827 generic.go:356] "Generic (PLEG): container finished" podID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerID="8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604" exitCode=0
Oct 29 23:03:14.706906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.706665    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" event={"ID":"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d","Type":"ContainerDied","Data":"8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604"}
Oct 29 23:03:14.707156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.707142    4827 scope.go:122] "RemoveContainer" containerID="8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604"
Oct 29 23:03:14.708754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.708728    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:14.708840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.708825    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:14.708883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.708870    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:15.002841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.995527    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:03:15.002841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:14.995591    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 29 23:03:15.630582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:15.630551    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:03:15.722700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:15.719338    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" event={"ID":"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d","Type":"ContainerStarted","Data":"2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590"}
Oct 29 23:03:16.002397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:16.002309    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:03:16.002650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:16.002617    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 29 23:03:16.538407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:16.538377    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:03:16.539806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:16.539782    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 29 23:03:16.539952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:16.539830    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 29 23:03:16.697795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:16.696953    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:03:17.046385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:17.046288    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:03:17.046385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:17.046351    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 29 23:03:17.538227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:17.538175    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 29 23:03:17.538455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:17.538257    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 29 23:03:18.008791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:17.999863    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:03:18.008791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:17.999920    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 29 23:03:18.538890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:18.538859    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 29 23:03:18.539298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:18.538913    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 29 23:03:19.006443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:19.004669    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:19.006443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:19.006212    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:19.010336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:19.010285    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:03:19.538256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:19.538227    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 29 23:03:19.538426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:19.538275    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 29 23:03:20.538652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:20.538615    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 29 23:03:20.539079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:20.538668    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 29 23:03:21.626803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:21.626731    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:03:21.627587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:21.627560    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:03:21.629427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:21.629394    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:03:28.626401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:28.626368    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"]
Oct 29 23:03:28.626867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:28.626569    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" containerID="cri-o://f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2" gracePeriod=30
Oct 29 23:03:30.442757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.442722    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"]
Oct 29 23:03:30.447671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.447649    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:30.464171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.464143    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-14debc\""
Oct 29 23:03:30.500953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.500913    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-14debc-dockercfg-mzlq4\""
Oct 29 23:03:30.669798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.626532    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96516365-f87e-456e-85dd-dcda19696285-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:30.669798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.626866    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72zb\" (UniqueName: \"kubernetes.io/projected/96516365-f87e-456e-85dd-dcda19696285-kube-api-access-r72zb\") pod \"payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:30.737000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.736903    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72zb\" (UniqueName: \"kubernetes.io/projected/96516365-f87e-456e-85dd-dcda19696285-kube-api-access-r72zb\") pod \"payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:30.755218 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.755173    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96516365-f87e-456e-85dd-dcda19696285-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:30.771804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.756591    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96516365-f87e-456e-85dd-dcda19696285-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:30.782802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.773408    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"]
Oct 29 23:03:30.909786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:30.907544    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72zb\" (UniqueName: \"kubernetes.io/projected/96516365-f87e-456e-85dd-dcda19696285-kube-api-access-r72zb\") pod \"payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:31.062148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:31.062059    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:31.419425 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:31.419380    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96516365_f87e_456e_85dd_dcda19696285.slice/crio-b8d63b7891564a0b9e41e9e68b41ae2e64dab72803103a1c11ae87598d0858a0 WatchSource:0}: Error finding container b8d63b7891564a0b9e41e9e68b41ae2e64dab72803103a1c11ae87598d0858a0: Status 404 returned error can't find the container with id b8d63b7891564a0b9e41e9e68b41ae2e64dab72803103a1c11ae87598d0858a0
Oct 29 23:03:31.420228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:31.420202    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"]
Oct 29 23:03:32.010340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:32.010307    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5" event={"ID":"96516365-f87e-456e-85dd-dcda19696285","Type":"ContainerStarted","Data":"2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3"}
Oct 29 23:03:32.010745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:32.010367    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5" event={"ID":"96516365-f87e-456e-85dd-dcda19696285","Type":"ContainerStarted","Data":"b8d63b7891564a0b9e41e9e68b41ae2e64dab72803103a1c11ae87598d0858a0"}
Oct 29 23:03:32.010745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:32.010393    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:32.763787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:32.762039    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fg5ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:32.763787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:32.762122    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:32.833532 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:32.833500    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:03:33.042191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:33.041868    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5" podStartSLOduration=3.041847752 podStartE2EDuration="3.041847752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:32.093904182 +0000 UTC m=+1247.716815204" watchObservedRunningTime="2026-10-29 23:03:33.041847752 +0000 UTC m=+1248.664758775"
Oct 29 23:03:33.047105 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:33.047001    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"]
Oct 29 23:03:33.059657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:33.059624    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9f72b8-fh5fw"]
Oct 29 23:03:33.947095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:33.947072    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 23:03:34.000273 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.000235    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-certs\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.000539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.000517    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4930d595-577e-4d00-805d-c04ed12523b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4930d595-577e-4d00-805d-c04ed12523b6-istiod-ca-cert\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.000735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.000575    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-certs" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:34.000897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.000880    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-envoy\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.001022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.001006    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-credential-socket\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.001166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.000926    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4930d595-577e-4d00-805d-c04ed12523b6-istiod-ca-cert" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:34.001323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.001306    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4930d595-577e-4d00-805d-c04ed12523b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4930d595-577e-4d00-805d-c04ed12523b6-istio-podinfo\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.001445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.001424    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-kube-api-access-9kh7s\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-kube-api-access-9kh7s\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.001521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.001503    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-istio-token\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-istio-token\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.001629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.001615    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-socket\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.001742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.001728    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-data\") pod \"4930d595-577e-4d00-805d-c04ed12523b6\" (UID: \"4930d595-577e-4d00-805d-c04ed12523b6\") "
Oct 29 23:03:34.002112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.001499    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-credential-socket" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:34.002347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.002327    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-socket" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:34.002544 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.002525    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-data" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:34.003499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.003479    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.003648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.003634    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.003846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.003831    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.003972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.003951    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.004064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.004054    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4930d595-577e-4d00-805d-c04ed12523b6-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.005232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.005191    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4930d595-577e-4d00-805d-c04ed12523b6-istio-podinfo" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:34.013245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.013219    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-envoy" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:34.013339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.013308    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-istio-token" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:34.014589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.014552    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-kube-api-access-9kh7s" pod "4930d595-577e-4d00-805d-c04ed12523b6" (UID: "4930d595-577e-4d00-805d-c04ed12523b6"). InnerVolumeSpecName "kube-api-access-9kh7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:34.058760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.058723    4827 generic.go:356] "Generic (PLEG): container finished" podID="4930d595-577e-4d00-805d-c04ed12523b6" containerID="f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2" exitCode=0
Oct 29 23:03:34.059272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.058800    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" event={"ID":"4930d595-577e-4d00-805d-c04ed12523b6","Type":"ContainerDied","Data":"f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2"}
Oct 29 23:03:34.059272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.058853    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws" event={"ID":"4930d595-577e-4d00-805d-c04ed12523b6","Type":"ContainerDied","Data":"845c4aa327938f7d0891f01dac1535b9d40a64f0fc5445b8298114bef51e4c9d"}
Oct 29 23:03:34.059272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.058885    4827 scope.go:122] "RemoveContainer" containerID="f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2"
Oct 29 23:03:34.059272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.058995    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"
Oct 29 23:03:34.073920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.073887    4827 scope.go:122] "RemoveContainer" containerID="f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2"
Oct 29 23:03:34.074208 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:34.074183    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2\": container with ID starting with f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2 not found: ID does not exist" containerID="f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2"
Oct 29 23:03:34.074291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.074220    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2"} err="failed to get container status \"f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2\": rpc error: code = NotFound desc = could not find container \"f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2\": container with ID starting with f4400aa8a407c1deafd32bd21bc6503402ad4568f9f351140aed4b7e11dbc8b2 not found: ID does not exist"
Oct 29 23:03:34.105919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.105895    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4930d595-577e-4d00-805d-c04ed12523b6-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.105919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.105918    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4930d595-577e-4d00-805d-c04ed12523b6-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.105919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.105927    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kh7s\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-kube-api-access-9kh7s\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.106192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.105937    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4930d595-577e-4d00-805d-c04ed12523b6-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:34.115033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.115011    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"]
Oct 29 23:03:34.119721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.119698    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fg5ws"]
Oct 29 23:03:34.490069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.490042    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4930d595-577e-4d00-805d-c04ed12523b6" path="/var/lib/kubelet/pods/4930d595-577e-4d00-805d-c04ed12523b6/volumes"
Oct 29 23:03:34.490609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:34.490592    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b735b8dd-105e-40c8-96eb-b5af28d4d387" path="/var/lib/kubelet/pods/b735b8dd-105e-40c8-96eb-b5af28d4d387/volumes"
Oct 29 23:03:36.040739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:36.040708    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"]
Oct 29 23:03:36.041218 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:36.040972    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy" containerID="cri-o://13e5ebd47abfca6199c0be3b3ec2aec8a7fcb848b3a4ba7d8cf20cb4e64ef93a" gracePeriod=30
Oct 29 23:03:36.048813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:36.046027    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"]
Oct 29 23:03:36.048813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:36.046248    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy" containerID="cri-o://ffa002f5bf5e820908b63e5a1115cdceab0a8bb010b29686ae8f86c271de6bb5" gracePeriod=30
Oct 29 23:03:37.417433 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.417390    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"]
Oct 29 23:03:37.418491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.418466    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy"
Oct 29 23:03:37.418491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.418490    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy"
Oct 29 23:03:37.418632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.418616    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4930d595-577e-4d00-805d-c04ed12523b6" containerName="istio-proxy"
Oct 29 23:03:37.424128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.424104    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.441050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.441028    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c73f1d-openshift-default-dockercfg-bvvcf\""
Oct 29 23:03:37.458565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.458535    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"]
Oct 29 23:03:37.534811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.534756    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5dacfa-5542-4a7d-a809-f02354a36829-istiod-ca-cert\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.534979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.534907    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-data\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.535055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.535039    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-credential-socket\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.535194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.535173    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzvl\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-kube-api-access-5vzvl\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.535283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.535265    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-socket\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.535411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.535394    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-envoy\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.535463 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.535448    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-podinfo\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.535589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.535577    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-token\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.535630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.535618    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-certs\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.614001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.613936    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"]
Oct 29 23:03:37.622131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.622089    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy" containerID="cri-o://26afd1d7da919b9e0a316d2788c6ba9a07361a0eb3d234da0d31c59585ec3043" gracePeriod=30
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644006    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-data\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644143    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-credential-socket\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644200    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzvl\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-kube-api-access-5vzvl\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644241    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-socket\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644352    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-envoy\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644403    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-podinfo\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644443    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-token\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644499    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-certs\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.645480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.644613    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5dacfa-5542-4a7d-a809-f02354a36829-istiod-ca-cert\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.677811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.677710    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-data\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.706918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.678557    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-socket\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.707814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.707781    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-credential-socket\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.721985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.721938    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-certs\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.794793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.781858    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5dacfa-5542-4a7d-a809-f02354a36829-istiod-ca-cert\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:37.966971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.947273    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"]
Oct 29 23:03:37.966971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:37.947542    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy" containerID="cri-o://8269de70e3b79770f9b2c8a2fcf24a8ce478130576fe0103217e86574710a6ed" gracePeriod=30
Oct 29 23:03:38.034453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.034371    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-envoy\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:38.035185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.035136    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-podinfo\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:38.051811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.039496    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-token\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:38.052823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.052798    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzvl\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-kube-api-access-5vzvl\") pod \"e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") " pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:38.348716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.348624    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:38.689500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.689476    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"]
Oct 29 23:03:38.692296 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:38.692270    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5dacfa_5542_4a7d_a809_f02354a36829.slice/crio-53258a0f5eb76b82956099e339bc1407dc274c616c38565d89dcf7b6f977676f WatchSource:0}: Error finding container 53258a0f5eb76b82956099e339bc1407dc274c616c38565d89dcf7b6f977676f: Status 404 returned error can't find the container with id 53258a0f5eb76b82956099e339bc1407dc274c616c38565d89dcf7b6f977676f
Oct 29 23:03:38.695553 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.695507    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:38.695644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.695584    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:38.695644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:38.695620    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:39.129159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.128877    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": read tcp 10.134.0.2:34450->10.134.0.167:15021: read: connection reset by peer" start-of-body=
Oct 29 23:03:39.129159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.128934    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": read tcp 10.134.0.2:34450->10.134.0.167:15021: read: connection reset by peer"
Oct 29 23:03:39.167117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.167086    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" event={"ID":"ad5dacfa-5542-4a7d-a809-f02354a36829","Type":"ContainerStarted","Data":"75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda"}
Oct 29 23:03:39.167297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.167147    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" event={"ID":"ad5dacfa-5542-4a7d-a809-f02354a36829","Type":"ContainerStarted","Data":"53258a0f5eb76b82956099e339bc1407dc274c616c38565d89dcf7b6f977676f"}
Oct 29 23:03:39.239136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.239083    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podStartSLOduration=2.239068358 podStartE2EDuration="2.239068358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:39.223926541 +0000 UTC m=+1254.846837563" watchObservedRunningTime="2026-10-29 23:03:39.239068358 +0000 UTC m=+1254.861979378"
Oct 29 23:03:39.349412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.349334    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:39.351005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.350975    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:03:39.351172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:39.351025    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:03:40.177575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:40.177543    4827 generic.go:356] "Generic (PLEG): container finished" podID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerID="7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da" exitCode=0
Oct 29 23:03:40.178086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:40.177611    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" event={"ID":"c0ba4855-3fc6-48d4-81da-448a9ff517af","Type":"ContainerDied","Data":"7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da"}
Oct 29 23:03:40.178086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:40.178035    4827 scope.go:122] "RemoveContainer" containerID="7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da"
Oct 29 23:03:40.179633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:40.179602    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:40.179746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:40.179690    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:40.179860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:40.179740    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:40.363123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:40.363091    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:41.282425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.282390    4827 generic.go:356] "Generic (PLEG): container finished" podID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerID="ffa002f5bf5e820908b63e5a1115cdceab0a8bb010b29686ae8f86c271de6bb5" exitCode=0
Oct 29 23:03:41.283129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.283103    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" event={"ID":"8803d40a-f432-4846-a0e3-ae79296ec8d6","Type":"ContainerDied","Data":"ffa002f5bf5e820908b63e5a1115cdceab0a8bb010b29686ae8f86c271de6bb5"}
Oct 29 23:03:41.301563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.296206    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" event={"ID":"c0ba4855-3fc6-48d4-81da-448a9ff517af","Type":"ContainerStarted","Data":"4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c"}
Oct 29 23:03:41.315418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.315385    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:41.332512 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.332433    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:03:41.404824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.384452    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:03:41.431927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.412928    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 29 23:03:41.431927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:41.412989    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 29 23:03:42.356841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.356140    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:03:42.365536 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.365126    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:03:42.401253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.398333    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 29 23:03:42.401253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.398397    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 29 23:03:42.402465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.401964    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:03:42.438072 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.438034    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:03:42.447226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.447192    4827 generic.go:356] "Generic (PLEG): container finished" podID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerID="13e5ebd47abfca6199c0be3b3ec2aec8a7fcb848b3a4ba7d8cf20cb4e64ef93a" exitCode=0
Oct 29 23:03:42.450706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.450672    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" event={"ID":"435e6749-83cf-4b6c-8d54-189ad5aeb204","Type":"ContainerDied","Data":"13e5ebd47abfca6199c0be3b3ec2aec8a7fcb848b3a4ba7d8cf20cb4e64ef93a"}
Oct 29 23:03:42.555374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.540484    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": context deadline exceeded"
Oct 29 23:03:42.686740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.686710    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 23:03:42.706957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.706926    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.750048    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-kube-api-access-9v5r4\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-kube-api-access-9v5r4\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.750174    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-credential-socket\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.750223    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-data\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.750278    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-kube-api-access-t2tk2\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-kube-api-access-t2tk2\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.750311    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-credential-socket\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.750372    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8803d40a-f432-4846-a0e3-ae79296ec8d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8803d40a-f432-4846-a0e3-ae79296ec8d6-istiod-ca-cert\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.750462    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-token\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-token\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.751429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.751135    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-data" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.754016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.753987    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-credential-socket" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.754167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754017    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-socket\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.754167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754089    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-socket\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.754167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754144    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-envoy\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.754337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754246    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435e6749-83cf-4b6c-8d54-189ad5aeb204-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435e6749-83cf-4b6c-8d54-189ad5aeb204-istiod-ca-cert\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.754337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754325    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-token\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-token\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.754453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754366    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-podinfo\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.754453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754424    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-certs\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.754562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754463    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-envoy\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.754562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754525    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-socket" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.754562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754541    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-certs\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.754866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754582    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-data\") pod \"8803d40a-f432-4846-a0e3-ae79296ec8d6\" (UID: \"8803d40a-f432-4846-a0e3-ae79296ec8d6\") "
Oct 29 23:03:42.754866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.754615    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-podinfo\") pod \"435e6749-83cf-4b6c-8d54-189ad5aeb204\" (UID: \"435e6749-83cf-4b6c-8d54-189ad5aeb204\") "
Oct 29 23:03:42.755542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.755505    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.755542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.755533    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.755711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.755557    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.756135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.756109    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-certs" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.756631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.756452    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-data" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.757050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.757019    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-socket" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.757231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.757205    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-certs" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.799931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.760581    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8803d40a-f432-4846-a0e3-ae79296ec8d6-istiod-ca-cert" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:42.799931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.760638    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-envoy" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.799931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.761039    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435e6749-83cf-4b6c-8d54-189ad5aeb204-istiod-ca-cert" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:42.800222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.799861    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-credential-socket" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.886106 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886065    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.886347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886326    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8803d40a-f432-4846-a0e3-ae79296ec8d6-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.886473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886459    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.886597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886580    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435e6749-83cf-4b6c-8d54-189ad5aeb204-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.886700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886688    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.886807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886793    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.886915 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886903    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.887002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.886993    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.943996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.943926    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-podinfo" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:42.990509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.990476    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.999893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:42.995067    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-podinfo" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:43.029474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.029440    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-token" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.029839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.029809    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-envoy" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.030057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.030041    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-kube-api-access-t2tk2" pod "8803d40a-f432-4846-a0e3-ae79296ec8d6" (UID: "8803d40a-f432-4846-a0e3-ae79296ec8d6"). InnerVolumeSpecName "kube-api-access-t2tk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.030248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.030233    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-kube-api-access-9v5r4" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "kube-api-access-9v5r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.039231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.039182    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-token" pod "435e6749-83cf-4b6c-8d54-189ad5aeb204" (UID: "435e6749-83cf-4b6c-8d54-189ad5aeb204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.056077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.056052    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:03:43.108869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.102706    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.111870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.111847    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.112021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.112004    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.112126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.112110    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v5r4\" (UniqueName: \"kubernetes.io/projected/435e6749-83cf-4b6c-8d54-189ad5aeb204-kube-api-access-9v5r4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.112219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.112207    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tk2\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-kube-api-access-t2tk2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.112386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.112372    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8803d40a-f432-4846-a0e3-ae79296ec8d6-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.449527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.449102    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 29 23:03:43.449527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.449156    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 29 23:03:43.505867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.503476    4827 generic.go:356] "Generic (PLEG): container finished" podID="567e72a9-9c4c-4522-81bc-479aac358efe" containerID="8269de70e3b79770f9b2c8a2fcf24a8ce478130576fe0103217e86574710a6ed" exitCode=0
Oct 29 23:03:43.505867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.503658    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" event={"ID":"567e72a9-9c4c-4522-81bc-479aac358efe","Type":"ContainerDied","Data":"8269de70e3b79770f9b2c8a2fcf24a8ce478130576fe0103217e86574710a6ed"}
Oct 29 23:03:43.506107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.506032    4827 generic.go:356] "Generic (PLEG): container finished" podID="1ad67055-0ab0-4997-b160-499db019ba1b" containerID="26afd1d7da919b9e0a316d2788c6ba9a07361a0eb3d234da0d31c59585ec3043" exitCode=0
Oct 29 23:03:43.506173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.506124    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" event={"ID":"1ad67055-0ab0-4997-b160-499db019ba1b","Type":"ContainerDied","Data":"26afd1d7da919b9e0a316d2788c6ba9a07361a0eb3d234da0d31c59585ec3043"}
Oct 29 23:03:43.513329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.513305    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq" event={"ID":"8803d40a-f432-4846-a0e3-ae79296ec8d6","Type":"ContainerDied","Data":"5402b9ccaaf2b66dd7536108ad323d08e1aec06a41b839f7a79400b9df8a2eec"}
Oct 29 23:03:43.513528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.513513    4827 scope.go:122] "RemoveContainer" containerID="ffa002f5bf5e820908b63e5a1115cdceab0a8bb010b29686ae8f86c271de6bb5"
Oct 29 23:03:43.514073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.514051    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"
Oct 29 23:03:43.521181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.521142    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"
Oct 29 23:03:43.521310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.521166    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp" event={"ID":"435e6749-83cf-4b6c-8d54-189ad5aeb204","Type":"ContainerDied","Data":"b9d162bd0743b060236485f7d003d47e963c3a3319112fb8125a0c4c939f0bdf"}
Oct 29 23:03:43.546883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.546858    4827 scope.go:122] "RemoveContainer" containerID="13e5ebd47abfca6199c0be3b3ec2aec8a7fcb848b3a4ba7d8cf20cb4e64ef93a"
Oct 29 23:03:43.611017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.610961    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"]
Oct 29 23:03:43.621355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.621326    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-5grvp"]
Oct 29 23:03:43.650008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.649981    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"]
Oct 29 23:03:43.696319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.696250    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-c85dq"]
Oct 29 23:03:43.698359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.698334    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 23:03:43.713122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.713096    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 23:03:43.736960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.736928    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"]
Oct 29 23:03:43.737574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737561    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy"
Oct 29 23:03:43.737713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737576    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy"
Oct 29 23:03:43.737713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737588    4827 cpu_manager.go:414] "containerMap: removing container" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy"
Oct 29 23:03:43.737713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737595    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy"
Oct 29 23:03:43.737713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737625    4827 cpu_manager.go:414] "containerMap: removing container" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy"
Oct 29 23:03:43.737713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737631    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy"
Oct 29 23:03:43.737713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737650    4827 cpu_manager.go:414] "containerMap: removing container" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy"
Oct 29 23:03:43.737713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737655    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy"
Oct 29 23:03:43.738039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737756    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" containerName="istio-proxy"
Oct 29 23:03:43.738039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737790    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" containerName="istio-proxy"
Oct 29 23:03:43.738039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737799    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" containerName="istio-proxy"
Oct 29 23:03:43.738039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.737807    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" containerName="istio-proxy"
Oct 29 23:03:43.740293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.740276    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:43.754565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.754539    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-istio-token\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-istio-token\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.754730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.754707    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-data\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.754857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.754840    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-socket\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.754936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.754886    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad67055-0ab0-4997-b160-499db019ba1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad67055-0ab0-4997-b160-499db019ba1b-istiod-ca-cert\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.755027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755007    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-data" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.755027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755019    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-kube-api-access-skgwh\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-kube-api-access-skgwh\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.755133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755095    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-certs\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.755133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755106    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-socket" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.755225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755153    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/567e72a9-9c4c-4522-81bc-479aac358efe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e72a9-9c4c-4522-81bc-479aac358efe-istio-podinfo\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.755281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755247    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-socket\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.755371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755342    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-certs" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.755432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755380    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad67055-0ab0-4997-b160-499db019ba1b-istiod-ca-cert" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:43.755432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755409    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-envoy\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.755556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755477    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-kube-api-access-ckgjk\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-kube-api-access-ckgjk\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.755556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755512    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-data\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.755638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755567    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-credential-socket\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.755638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755599    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-envoy\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.755638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755626    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ad67055-0ab0-4997-b160-499db019ba1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad67055-0ab0-4997-b160-499db019ba1b-istio-podinfo\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.755860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755656    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-certs\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.755860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755691    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-istio-token\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-istio-token\") pod \"1ad67055-0ab0-4997-b160-499db019ba1b\" (UID: \"1ad67055-0ab0-4997-b160-499db019ba1b\") "
Oct 29 23:03:43.755860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755797    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-credential-socket\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.755997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.755920    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567e72a9-9c4c-4522-81bc-479aac358efe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e72a9-9c4c-4522-81bc-479aac358efe-istiod-ca-cert\") pod \"567e72a9-9c4c-4522-81bc-479aac358efe\" (UID: \"567e72a9-9c4c-4522-81bc-479aac358efe\") "
Oct 29 23:03:43.756030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756006    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-credential-socket" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.756428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756406    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-certs" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.756638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756615    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2xz\" (UniqueName: \"kubernetes.io/projected/186d20be-496b-4a16-ac9b-f5869c365c0e-kube-api-access-pg2xz\") pod \"payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:43.756702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756666    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567e72a9-9c4c-4522-81bc-479aac358efe-istiod-ca-cert" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:43.756702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756686    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186d20be-496b-4a16-ac9b-f5869c365c0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:43.756884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756867    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.756999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756890    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.756999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756949    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.756999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756963    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e72a9-9c4c-4522-81bc-479aac358efe-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.756999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756975    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.756999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756986    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.756999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756988    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-socket" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.757298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.756998    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad67055-0ab0-4997-b160-499db019ba1b-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.761358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.761317    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-istio-token" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.761621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.761602    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c73f1d\""
Oct 29 23:03:43.761836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.761818    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c73f1d-dockercfg-lk9h9\""
Oct 29 23:03:43.762170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.762147    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-credential-socket" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.794123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.762857    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-istio-token" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.794123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.794045    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-data" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.796172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.796149    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"]
Oct 29 23:03:43.796931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.796908    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-kube-api-access-skgwh" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "kube-api-access-skgwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.797008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.796976    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-envoy" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.797399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.797357    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/567e72a9-9c4c-4522-81bc-479aac358efe-istio-podinfo" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:43.797615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.797594    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-kube-api-access-ckgjk" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "kube-api-access-ckgjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:43.797893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.797867    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-envoy" pod "567e72a9-9c4c-4522-81bc-479aac358efe" (UID: "567e72a9-9c4c-4522-81bc-479aac358efe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:43.799380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.799352    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ad67055-0ab0-4997-b160-499db019ba1b-istio-podinfo" pod "1ad67055-0ab0-4997-b160-499db019ba1b" (UID: "1ad67055-0ab0-4997-b160-499db019ba1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:43.848403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.848366    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"]
Oct 29 23:03:43.852540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.852513    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:43.857919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.857892    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2xz\" (UniqueName: \"kubernetes.io/projected/186d20be-496b-4a16-ac9b-f5869c365c0e-kube-api-access-pg2xz\") pod \"payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:43.858026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.857955    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186d20be-496b-4a16-ac9b-f5869c365c0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:43.858248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858230    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8q5l\" (UniqueName: \"kubernetes.io/projected/fd2f5612-2e40-420a-9648-9d62b745a0f8-kube-api-access-b8q5l\") pod \"payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:43.858327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858312    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2f5612-2e40-420a-9648-9d62b745a0f8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:43.858542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858517    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skgwh\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-kube-api-access-skgwh\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858548    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e72a9-9c4c-4522-81bc-479aac358efe-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858563    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858577    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858592    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckgjk\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-kube-api-access-ckgjk\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858604    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858617    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad67055-0ab0-4997-b160-499db019ba1b-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858621    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186d20be-496b-4a16-ac9b-f5869c365c0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858629    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad67055-0ab0-4997-b160-499db019ba1b-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858642    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad67055-0ab0-4997-b160-499db019ba1b-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858656    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e72a9-9c4c-4522-81bc-479aac358efe-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.858673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.858669    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e72a9-9c4c-4522-81bc-479aac358efe-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:43.897379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.897335    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"]
Oct 29 23:03:43.898113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.897741    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2xz\" (UniqueName: \"kubernetes.io/projected/186d20be-496b-4a16-ac9b-f5869c365c0e-kube-api-access-pg2xz\") pod \"payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:43.960430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.960392    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8q5l\" (UniqueName: \"kubernetes.io/projected/fd2f5612-2e40-420a-9648-9d62b745a0f8-kube-api-access-b8q5l\") pod \"payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:43.960723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.960704    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2f5612-2e40-420a-9648-9d62b745a0f8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:43.970008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:43.969978    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2f5612-2e40-420a-9648-9d62b745a0f8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:44.031467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.031435    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8q5l\" (UniqueName: \"kubernetes.io/projected/fd2f5612-2e40-420a-9648-9d62b745a0f8-kube-api-access-b8q5l\") pod \"payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:44.076868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.076833    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:44.176164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.176079    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:44.364166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.364133    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 29 23:03:44.364434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.364407    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 29 23:03:44.507235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.507206    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435e6749-83cf-4b6c-8d54-189ad5aeb204" path="/var/lib/kubelet/pods/435e6749-83cf-4b6c-8d54-189ad5aeb204/volumes"
Oct 29 23:03:44.507809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.507792    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8803d40a-f432-4846-a0e3-ae79296ec8d6" path="/var/lib/kubelet/pods/8803d40a-f432-4846-a0e3-ae79296ec8d6/volumes"
Oct 29 23:03:44.540282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.540262    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"
Oct 29 23:03:44.540612 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.540394    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp" event={"ID":"567e72a9-9c4c-4522-81bc-479aac358efe","Type":"ContainerDied","Data":"3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a"}
Oct 29 23:03:44.540612 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.540464    4827 scope.go:122] "RemoveContainer" containerID="8269de70e3b79770f9b2c8a2fcf24a8ce478130576fe0103217e86574710a6ed"
Oct 29 23:03:44.544442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.544255    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"
Oct 29 23:03:44.544442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.544253    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7" event={"ID":"1ad67055-0ab0-4997-b160-499db019ba1b","Type":"ContainerDied","Data":"111ff9556758f534b5af86f13bb6b32be314d7c01b1b559d4b36f3236e0eb5f1"}
Oct 29 23:03:44.763308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.763070    4827 scope.go:122] "RemoveContainer" containerID="26afd1d7da919b9e0a316d2788c6ba9a07361a0eb3d234da0d31c59585ec3043"
Oct 29 23:03:44.846012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.845982    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"]
Oct 29 23:03:44.858231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.858205    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lkfq7"]
Oct 29 23:03:44.900913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.900880    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"]
Oct 29 23:03:44.906166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.906138    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bkwfp"]
Oct 29 23:03:44.994226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.994100    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"]
Oct 29 23:03:44.998155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:44.998132    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"]
Oct 29 23:03:45.000096 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:45.000064    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186d20be_496b_4a16_ac9b_f5869c365c0e.slice/crio-ff785a604164a68eb2a4dedffa670583010ccaf7278ee5f762685f4401b4edb7 WatchSource:0}: Error finding container ff785a604164a68eb2a4dedffa670583010ccaf7278ee5f762685f4401b4edb7: Status 404 returned error can't find the container with id ff785a604164a68eb2a4dedffa670583010ccaf7278ee5f762685f4401b4edb7
Oct 29 23:03:45.002085 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:45.002060    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2f5612_2e40_420a_9648_9d62b745a0f8.slice/crio-e38f3d38018c585e332bdb638f4947578807e1704642b86e6fbc69979796dee7 WatchSource:0}: Error finding container e38f3d38018c585e332bdb638f4947578807e1704642b86e6fbc69979796dee7: Status 404 returned error can't find the container with id e38f3d38018c585e332bdb638f4947578807e1704642b86e6fbc69979796dee7
Oct 29 23:03:45.376065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.376025    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 29 23:03:45.376262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.376106    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 29 23:03:45.843225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.843137    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl" event={"ID":"186d20be-496b-4a16-ac9b-f5869c365c0e","Type":"ContainerStarted","Data":"7b808967477fcc0e505f66bacee7b687d76518c0ed5c2f540e0f7445d7da540f"}
Oct 29 23:03:45.843225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.843185    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl" event={"ID":"186d20be-496b-4a16-ac9b-f5869c365c0e","Type":"ContainerStarted","Data":"ff785a604164a68eb2a4dedffa670583010ccaf7278ee5f762685f4401b4edb7"}
Oct 29 23:03:45.843846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.843229    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:45.845175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.845149    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc" event={"ID":"fd2f5612-2e40-420a-9648-9d62b745a0f8","Type":"ContainerStarted","Data":"5f20810166f5bbd25dcf4472039af0caaff36ca42987084913a48e79569e8987"}
Oct 29 23:03:45.845298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.845197    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc" event={"ID":"fd2f5612-2e40-420a-9648-9d62b745a0f8","Type":"ContainerStarted","Data":"e38f3d38018c585e332bdb638f4947578807e1704642b86e6fbc69979796dee7"}
Oct 29 23:03:45.845298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.845225    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:45.908344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.908300    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl" podStartSLOduration=2.908287205 podStartE2EDuration="2.908287205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:45.904860085 +0000 UTC m=+1261.527771107" watchObservedRunningTime="2026-10-29 23:03:45.908287205 +0000 UTC m=+1261.531198225"
Oct 29 23:03:45.947120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:45.947069    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc" podStartSLOduration=2.947052173 podStartE2EDuration="2.947052173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:45.944201605 +0000 UTC m=+1261.567112627" watchObservedRunningTime="2026-10-29 23:03:45.947052173 +0000 UTC m=+1261.569963195"
Oct 29 23:03:46.366812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:46.363741    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 29 23:03:46.366812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:46.363813    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 29 23:03:46.496620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:46.496579    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad67055-0ab0-4997-b160-499db019ba1b" path="/var/lib/kubelet/pods/1ad67055-0ab0-4997-b160-499db019ba1b/volumes"
Oct 29 23:03:46.497265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:46.497242    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567e72a9-9c4c-4522-81bc-479aac358efe" path="/var/lib/kubelet/pods/567e72a9-9c4c-4522-81bc-479aac358efe/volumes"
Oct 29 23:03:47.384610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:47.384165    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 29 23:03:47.384610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:47.384225    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 29 23:03:47.737249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:47.737169    4827 scope.go:122] "RemoveContainer" containerID="b52ca8b7d95ce95c0118b453d0c5be467171cacccae9aa3f5b7b0f956b53854f"
Oct 29 23:03:47.748174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:47.748156    4827 scope.go:122] "RemoveContainer" containerID="b809d103950e410acc1464a4c6391eade1d23622cc359b8c9a4772949bf563d6"
Oct 29 23:03:47.758449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:47.758429    4827 scope.go:122] "RemoveContainer" containerID="d968acade4df5849b85b5c6a07b26aacc05e67d1a20cf38f5a925244046dfbd8"
Oct 29 23:03:47.776790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:47.775888    4827 scope.go:122] "RemoveContainer" containerID="09443086359dd15936c027e3c87a4e65a22e267c5ed3d7076b5e85d2f70b3216"
Oct 29 23:03:47.803527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:47.803505    4827 scope.go:122] "RemoveContainer" containerID="fb947ac7c49708d9166c7d556175d8119b4cf7789f6ddbc807309989a31c73e9"
Oct 29 23:03:48.510415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:48.510384    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:03:48.511082 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:48.510808    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:03:48.512276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:48.512253    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:03:49.744363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:49.744328    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"]
Oct 29 23:03:49.750950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:49.750928    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:49.767728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:49.767697    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"]
Oct 29 23:03:49.775947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:49.775578    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b7950\""
Oct 29 23:03:49.777031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:49.776199    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b7950-dockercfg-jr7tf\""
Oct 29 23:03:49.860272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:49.860231    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7c5692-52b1-4d25-88fc-4d731525aaeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:49.863787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:49.860588    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6n24\" (UniqueName: \"kubernetes.io/projected/3b7c5692-52b1-4d25-88fc-4d731525aaeb-kube-api-access-j6n24\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:50.079841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.079018    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n24\" (UniqueName: \"kubernetes.io/projected/3b7c5692-52b1-4d25-88fc-4d731525aaeb-kube-api-access-j6n24\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:50.079841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.079182    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7c5692-52b1-4d25-88fc-4d731525aaeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:50.085862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.080372    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7c5692-52b1-4d25-88fc-4d731525aaeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:50.129193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.128031    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"]
Oct 29 23:03:50.153464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.149969    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"]
Oct 29 23:03:50.153464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.150095    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.158826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.158797    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6n24\" (UniqueName: \"kubernetes.io/projected/3b7c5692-52b1-4d25-88fc-4d731525aaeb-kube-api-access-j6n24\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:50.287156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.287122    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrft\" (UniqueName: \"kubernetes.io/projected/333138a6-55c4-459d-89d2-f6044d08a481-kube-api-access-ltrft\") pod \"payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.287380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.287360    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333138a6-55c4-459d-89d2-f6044d08a481-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.364506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.364430    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:50.388616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.388586    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrft\" (UniqueName: \"kubernetes.io/projected/333138a6-55c4-459d-89d2-f6044d08a481-kube-api-access-ltrft\") pod \"payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.388881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.388852    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333138a6-55c4-459d-89d2-f6044d08a481-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.389507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.389489    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333138a6-55c4-459d-89d2-f6044d08a481-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.411911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.411877    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrft\" (UniqueName: \"kubernetes.io/projected/333138a6-55c4-459d-89d2-f6044d08a481-kube-api-access-ltrft\") pod \"payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.475873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.475837    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:50.769248 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:50.769181    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7c5692_52b1_4d25_88fc_4d731525aaeb.slice/crio-17ce362df0867185af11ab6c0361fadc2bcc77f670b0fd08fddb2eb55aec0727 WatchSource:0}: Error finding container 17ce362df0867185af11ab6c0361fadc2bcc77f670b0fd08fddb2eb55aec0727: Status 404 returned error can't find the container with id 17ce362df0867185af11ab6c0361fadc2bcc77f670b0fd08fddb2eb55aec0727
Oct 29 23:03:50.877789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.866854    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"]
Oct 29 23:03:50.937950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.929541    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"]
Oct 29 23:03:50.967066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.955392    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"]
Oct 29 23:03:50.967066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.955510    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:50.974751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.974719    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"]
Oct 29 23:03:50.998941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:50.991018    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-182195-openshift-default-dockercfg-pvbwp\""
Oct 29 23:03:51.179899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.179865    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-token\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180110    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istiod-ca-cert\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180209    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-credential-socket\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180376    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-socket\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180437    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-envoy\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180544 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180535    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-podinfo\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180599    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bst2h\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-kube-api-access-bst2h\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180688    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-certs\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.180804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.180793    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-data\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.203195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.203163    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm" event={"ID":"333138a6-55c4-459d-89d2-f6044d08a481","Type":"ContainerStarted","Data":"b62aaf4fb7129b930a5c1aff78d55cadd82c09785383c4135967a03752835ea6"}
Oct 29 23:03:51.204725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.204677    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7" event={"ID":"3b7c5692-52b1-4d25-88fc-4d731525aaeb","Type":"ContainerStarted","Data":"17ce362df0867185af11ab6c0361fadc2bcc77f670b0fd08fddb2eb55aec0727"}
Oct 29 23:03:51.282074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282049    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-podinfo\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282082    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst2h\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-kube-api-access-bst2h\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282138    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-certs\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282192    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-data\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282279    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-token\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282333    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istiod-ca-cert\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282377    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-credential-socket\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282536 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282521    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-socket\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282598 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282539    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-envoy\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282676 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282641    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-certs\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.282965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.282945    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-socket\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.283057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.283005    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-credential-socket\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.283199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.283175    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-data\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.283591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.283568    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istiod-ca-cert\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.285000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.284978    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-envoy\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.285199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.285179    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-podinfo\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.306009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.305986    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bst2h\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-kube-api-access-bst2h\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.306127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.306055    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-token\") pod \"e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") " pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.503934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.503905    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:51.823890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.822557    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"]
Oct 29 23:03:51.823890 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:51.823472    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98f78c6_6603_4c29_9b23_721bacf6d8d8.slice/crio-d278260aec771fdb244de7dfd1ec6ce11a96031a9f09adee72ce601908efb424 WatchSource:0}: Error finding container d278260aec771fdb244de7dfd1ec6ce11a96031a9f09adee72ce601908efb424: Status 404 returned error can't find the container with id d278260aec771fdb244de7dfd1ec6ce11a96031a9f09adee72ce601908efb424
Oct 29 23:03:51.827183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.827147    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:51.827283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.827235    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:51.827283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:51.827276    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:03:52.243815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.229997    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm" event={"ID":"333138a6-55c4-459d-89d2-f6044d08a481","Type":"ContainerStarted","Data":"9fa0da9b859d641517e405bfc1cc5db650b44fb2f63c6f26ccce383360e03eb1"}
Oct 29 23:03:52.243815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.230842    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:03:52.243815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.233576    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" event={"ID":"c98f78c6-6603-4c29-9b23-721bacf6d8d8","Type":"ContainerStarted","Data":"7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca"}
Oct 29 23:03:52.243815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.233630    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" event={"ID":"c98f78c6-6603-4c29-9b23-721bacf6d8d8","Type":"ContainerStarted","Data":"d278260aec771fdb244de7dfd1ec6ce11a96031a9f09adee72ce601908efb424"}
Oct 29 23:03:52.243815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.237011    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7" event={"ID":"3b7c5692-52b1-4d25-88fc-4d731525aaeb","Type":"ContainerStarted","Data":"1338acec613e3a575595d88c776eafa82bed1ff05db5fe41f2363c04bf3dd0a1"}
Oct 29 23:03:52.243815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.237834    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:03:52.298618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.298560    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm" podStartSLOduration=3.298540111 podStartE2EDuration="3.298540111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:52.280563235 +0000 UTC m=+1267.903474255" watchObservedRunningTime="2026-10-29 23:03:52.298540111 +0000 UTC m=+1267.921451133"
Oct 29 23:03:52.374632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.374573    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podStartSLOduration=2.3745538760000002 podStartE2EDuration="2.374553876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:52.328529254 +0000 UTC m=+1267.951440277" watchObservedRunningTime="2026-10-29 23:03:52.374553876 +0000 UTC m=+1267.997464896"
Oct 29 23:03:52.375217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.375179    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7" podStartSLOduration=3.375169356 podStartE2EDuration="3.375169356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:52.368167059 +0000 UTC m=+1267.991078079" watchObservedRunningTime="2026-10-29 23:03:52.375169356 +0000 UTC m=+1267.998080378"
Oct 29 23:03:52.504503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.504391    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:52.505953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.505922    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:03:52.506098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:52.505980    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:03:53.539791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:53.537621    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:54.092709 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:54.092670    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:03:54.251295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:54.251268    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:54.252284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:54.252266    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:03:54.883842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:54.883806    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"]
Oct 29 23:03:54.884327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:54.884038    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j" podUID="0f12977a-6a23-47a5-be17-6af3c5b73688" containerName="maas-api" containerID="cri-o://9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921" gracePeriod=30
Oct 29 23:03:55.056691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.056657    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"]
Oct 29 23:03:55.056926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.056903    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw" podUID="08cb891c-a60e-4593-8b63-94dc2bd1924f" containerName="payload-processing" containerID="cri-o://1ae77672410e4a707dd6ed4e9867fd59574591807504684a0662db77d2a66e69" gracePeriod=30
Oct 29 23:03:55.205681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.205653    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:03:55.206126 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:55.206074    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:03:55.272188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.271369    4827 generic.go:356] "Generic (PLEG): container finished" podID="0f12977a-6a23-47a5-be17-6af3c5b73688" containerID="9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921" exitCode=0
Oct 29 23:03:55.272188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.271547    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j" event={"ID":"0f12977a-6a23-47a5-be17-6af3c5b73688","Type":"ContainerDied","Data":"9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921"}
Oct 29 23:03:55.272188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.271626    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j" event={"ID":"0f12977a-6a23-47a5-be17-6af3c5b73688","Type":"ContainerDied","Data":"4d30d7ccb52ee09bd09d7fffd829aa28aa755e9d49b18c66682f9c91b784e7cb"}
Oct 29 23:03:55.272188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.271669    4827 scope.go:122] "RemoveContainer" containerID="9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921"
Oct 29 23:03:55.272188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.271874    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"
Oct 29 23:03:55.275671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.274981    4827 generic.go:356] "Generic (PLEG): container finished" podID="08cb891c-a60e-4593-8b63-94dc2bd1924f" containerID="1ae77672410e4a707dd6ed4e9867fd59574591807504684a0662db77d2a66e69" exitCode=0
Oct 29 23:03:55.281781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.275871    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw" event={"ID":"08cb891c-a60e-4593-8b63-94dc2bd1924f","Type":"ContainerDied","Data":"1ae77672410e4a707dd6ed4e9867fd59574591807504684a0662db77d2a66e69"}
Oct 29 23:03:55.297542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.297519    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls\") pod \"0f12977a-6a23-47a5-be17-6af3c5b73688\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") "
Oct 29 23:03:55.297641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.297610    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f12977a-6a23-47a5-be17-6af3c5b73688-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f12977a-6a23-47a5-be17-6af3c5b73688-openshift-service-ca\") pod \"0f12977a-6a23-47a5-be17-6af3c5b73688\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") "
Oct 29 23:03:55.297827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.297811    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-metrics-certs\") pod \"0f12977a-6a23-47a5-be17-6af3c5b73688\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") "
Oct 29 23:03:55.297903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.297842    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f12977a-6a23-47a5-be17-6af3c5b73688-kube-api-access-lz6jr\" (UniqueName: \"kubernetes.io/projected/0f12977a-6a23-47a5-be17-6af3c5b73688-kube-api-access-lz6jr\") pod \"0f12977a-6a23-47a5-be17-6af3c5b73688\" (UID: \"0f12977a-6a23-47a5-be17-6af3c5b73688\") "
Oct 29 23:03:55.298935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.298915    4827 scope.go:122] "RemoveContainer" containerID="9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921"
Oct 29 23:03:55.299304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.299276    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f12977a-6a23-47a5-be17-6af3c5b73688-openshift-service-ca" pod "0f12977a-6a23-47a5-be17-6af3c5b73688" (UID: "0f12977a-6a23-47a5-be17-6af3c5b73688"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:55.299581 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:03:55.299545    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921\": container with ID starting with 9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921 not found: ID does not exist" containerID="9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921"
Oct 29 23:03:55.299644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.299591    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921"} err="failed to get container status \"9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921\": rpc error: code = NotFound desc = could not find container \"9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921\": container with ID starting with 9964ad40d1319d0963bab99b20c253057634e691ec5c24c4efe916d9fa2db921 not found: ID does not exist"
Oct 29 23:03:55.303993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.302867    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f12977a-6a23-47a5-be17-6af3c5b73688-kube-api-access-lz6jr" pod "0f12977a-6a23-47a5-be17-6af3c5b73688" (UID: "0f12977a-6a23-47a5-be17-6af3c5b73688"). InnerVolumeSpecName "kube-api-access-lz6jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:55.303993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.302946    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-metrics-certs" pod "0f12977a-6a23-47a5-be17-6af3c5b73688" (UID: "0f12977a-6a23-47a5-be17-6af3c5b73688"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:55.304547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.304520    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls" pod "0f12977a-6a23-47a5-be17-6af3c5b73688" (UID: "0f12977a-6a23-47a5-be17-6af3c5b73688"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:55.346591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.346570    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:03:55.436269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.436173    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cb891c-a60e-4593-8b63-94dc2bd1924f-kube-api-access-24szf\" (UniqueName: \"kubernetes.io/projected/08cb891c-a60e-4593-8b63-94dc2bd1924f-kube-api-access-24szf\") pod \"08cb891c-a60e-4593-8b63-94dc2bd1924f\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") "
Oct 29 23:03:55.437215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.437188    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08cb891c-a60e-4593-8b63-94dc2bd1924f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cb891c-a60e-4593-8b63-94dc2bd1924f-plugins-config-volume\") pod \"08cb891c-a60e-4593-8b63-94dc2bd1924f\" (UID: \"08cb891c-a60e-4593-8b63-94dc2bd1924f\") "
Oct 29 23:03:55.471791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.438581    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:55.472305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.472281    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f12977a-6a23-47a5-be17-6af3c5b73688-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:55.476875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.476844    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f12977a-6a23-47a5-be17-6af3c5b73688-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:55.477057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.477036    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz6jr\" (UniqueName: \"kubernetes.io/projected/0f12977a-6a23-47a5-be17-6af3c5b73688-kube-api-access-lz6jr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:55.484840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.457988    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cb891c-a60e-4593-8b63-94dc2bd1924f-kube-api-access-24szf" pod "08cb891c-a60e-4593-8b63-94dc2bd1924f" (UID: "08cb891c-a60e-4593-8b63-94dc2bd1924f"). InnerVolumeSpecName "kube-api-access-24szf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:55.485024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.472224    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cb891c-a60e-4593-8b63-94dc2bd1924f-plugins-config-volume" pod "08cb891c-a60e-4593-8b63-94dc2bd1924f" (UID: "08cb891c-a60e-4593-8b63-94dc2bd1924f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:55.622793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.618870    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24szf\" (UniqueName: \"kubernetes.io/projected/08cb891c-a60e-4593-8b63-94dc2bd1924f-kube-api-access-24szf\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:55.622793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.618905    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cb891c-a60e-4593-8b63-94dc2bd1924f-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:03:55.749842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.740061    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"]
Oct 29 23:03:55.799199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.799154    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"]
Oct 29 23:03:55.804450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:55.804310    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f173b1-7fc5b54458-vkz6j"]
Oct 29 23:03:56.285424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.285398    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"
Oct 29 23:03:56.285424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.285410    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw" event={"ID":"08cb891c-a60e-4593-8b63-94dc2bd1924f","Type":"ContainerDied","Data":"d11c93dfd0306a31f75f7906402779f40fa7f9658c7339454d4833dc3102515f"}
Oct 29 23:03:56.285898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.285474    4827 scope.go:122] "RemoveContainer" containerID="1ae77672410e4a707dd6ed4e9867fd59574591807504684a0662db77d2a66e69"
Oct 29 23:03:56.363800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.354945    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"]
Oct 29 23:03:56.412788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.403924    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f173b1-55c55b9cd-7bbsw"]
Oct 29 23:03:56.529793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.529198    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08cb891c-a60e-4593-8b63-94dc2bd1924f" path="/var/lib/kubelet/pods/08cb891c-a60e-4593-8b63-94dc2bd1924f/volumes"
Oct 29 23:03:56.546847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.537205    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f12977a-6a23-47a5-be17-6af3c5b73688" path="/var/lib/kubelet/pods/0f12977a-6a23-47a5-be17-6af3c5b73688/volumes"
Oct 29 23:03:56.856133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.856060    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:03:56.856133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:56.856123    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:03:57.373052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.373024    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"]
Oct 29 23:03:57.373706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.373692    4827 cpu_manager.go:414] "containerMap: removing container" podUID="08cb891c-a60e-4593-8b63-94dc2bd1924f" containerName="payload-processing"
Oct 29 23:03:57.373752 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.373708    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cb891c-a60e-4593-8b63-94dc2bd1924f" containerName="payload-processing"
Oct 29 23:03:57.373752 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.373722    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0f12977a-6a23-47a5-be17-6af3c5b73688" containerName="maas-api"
Oct 29 23:03:57.373752 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.373727    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f12977a-6a23-47a5-be17-6af3c5b73688" containerName="maas-api"
Oct 29 23:03:57.373874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.373838    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cb891c-a60e-4593-8b63-94dc2bd1924f" containerName="payload-processing"
Oct 29 23:03:57.373874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.373850    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f12977a-6a23-47a5-be17-6af3c5b73688" containerName="maas-api"
Oct 29 23:03:57.378747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.378725    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.393957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.393926    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-182195\""
Oct 29 23:03:57.393957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.393930    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-182195-dockercfg-24c4t\""
Oct 29 23:03:57.396220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.396198    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"]
Oct 29 23:03:57.650614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.650480    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"]
Oct 29 23:03:57.707789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.669714    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqts\" (UniqueName: \"kubernetes.io/projected/5568ec3c-2358-4041-8261-7bb8de202ee0-kube-api-access-khqts\") pod \"payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.707789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.687571    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5568ec3c-2358-4041-8261-7bb8de202ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.782992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.775939    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"]
Oct 29 23:03:57.782992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.776088    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:57.803417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.790233    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqts\" (UniqueName: \"kubernetes.io/projected/5568ec3c-2358-4041-8261-7bb8de202ee0-kube-api-access-khqts\") pod \"payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.803417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.790281    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5568ec3c-2358-4041-8261-7bb8de202ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.803417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.791504    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5568ec3c-2358-4041-8261-7bb8de202ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.814803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.814743    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqts\" (UniqueName: \"kubernetes.io/projected/5568ec3c-2358-4041-8261-7bb8de202ee0-kube-api-access-khqts\") pod \"payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.891642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.891613    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrzr\" (UniqueName: \"kubernetes.io/projected/be92200b-d421-4f11-9a37-e7c586554482-kube-api-access-blrzr\") pod \"payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:57.891866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.891752    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92200b-d421-4f11-9a37-e7c586554482-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:57.993307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.993211    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrzr\" (UniqueName: \"kubernetes.io/projected/be92200b-d421-4f11-9a37-e7c586554482-kube-api-access-blrzr\") pod \"payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:57.993307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.993281    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92200b-d421-4f11-9a37-e7c586554482-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:57.993546 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.993337    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:57.994045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:57.994027    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92200b-d421-4f11-9a37-e7c586554482-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:58.016413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:58.016384    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrzr\" (UniqueName: \"kubernetes.io/projected/be92200b-d421-4f11-9a37-e7c586554482-kube-api-access-blrzr\") pod \"payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:58.117337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:58.117304    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:58.308513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:58.308475    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"]
Oct 29 23:03:58.308858 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:58.308833    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5568ec3c_2358_4041_8261_7bb8de202ee0.slice/crio-eef155d296ad0e7e73f8df3ba9c3f197f4f3116bc983cf8283d97cb1755a2a35 WatchSource:0}: Error finding container eef155d296ad0e7e73f8df3ba9c3f197f4f3116bc983cf8283d97cb1755a2a35: Status 404 returned error can't find the container with id eef155d296ad0e7e73f8df3ba9c3f197f4f3116bc983cf8283d97cb1755a2a35
Oct 29 23:03:58.430483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:58.430454    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"]
Oct 29 23:03:58.430908 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:03:58.430877    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe92200b_d421_4f11_9a37_e7c586554482.slice/crio-934f76f78c1fbf5548bdee8aff16c62c0b0a38830112461a5dc4db10c611a6c2 WatchSource:0}: Error finding container 934f76f78c1fbf5548bdee8aff16c62c0b0a38830112461a5dc4db10c611a6c2: Status 404 returned error can't find the container with id 934f76f78c1fbf5548bdee8aff16c62c0b0a38830112461a5dc4db10c611a6c2
Oct 29 23:03:59.313845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.313803    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk" event={"ID":"5568ec3c-2358-4041-8261-7bb8de202ee0","Type":"ContainerStarted","Data":"0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3"}
Oct 29 23:03:59.314065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.313864    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk" event={"ID":"5568ec3c-2358-4041-8261-7bb8de202ee0","Type":"ContainerStarted","Data":"eef155d296ad0e7e73f8df3ba9c3f197f4f3116bc983cf8283d97cb1755a2a35"}
Oct 29 23:03:59.314065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.313887    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:03:59.315661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.315637    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk" event={"ID":"be92200b-d421-4f11-9a37-e7c586554482","Type":"ContainerStarted","Data":"4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414"}
Oct 29 23:03:59.315793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.315688    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk" event={"ID":"be92200b-d421-4f11-9a37-e7c586554482","Type":"ContainerStarted","Data":"934f76f78c1fbf5548bdee8aff16c62c0b0a38830112461a5dc4db10c611a6c2"}
Oct 29 23:03:59.315793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.315719    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:03:59.359672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.359609    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk" podStartSLOduration=2.359590375 podStartE2EDuration="2.359590375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:59.353861921 +0000 UTC m=+1274.976772943" watchObservedRunningTime="2026-10-29 23:03:59.359590375 +0000 UTC m=+1274.982501394"
Oct 29 23:03:59.396603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:03:59.396549    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk" podStartSLOduration=2.396534324 podStartE2EDuration="2.396534324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:59.393188566 +0000 UTC m=+1275.016099586" watchObservedRunningTime="2026-10-29 23:03:59.396534324 +0000 UTC m=+1275.019445387"
Oct 29 23:04:00.799813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:00.799746    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw" podUID="88b8747f-33a6-4b68-9de1-e079a5ee7ac7" containerName="payload-pre-processing" containerID="cri-o://f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4" gracePeriod=25
Oct 29 23:04:01.091821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.091795    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:04:01.162245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.162211    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-kube-api-access-l29ss\" (UniqueName: \"kubernetes.io/projected/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-kube-api-access-l29ss\") pod \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") "
Oct 29 23:04:01.162442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.162404    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-plugins-config-volume\") pod \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\" (UID: \"88b8747f-33a6-4b68-9de1-e079a5ee7ac7\") "
Oct 29 23:04:01.162857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.162830    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-plugins-config-volume" pod "88b8747f-33a6-4b68-9de1-e079a5ee7ac7" (UID: "88b8747f-33a6-4b68-9de1-e079a5ee7ac7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:01.163082 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.163067    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:01.164427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.164396    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-kube-api-access-l29ss" pod "88b8747f-33a6-4b68-9de1-e079a5ee7ac7" (UID: "88b8747f-33a6-4b68-9de1-e079a5ee7ac7"). InnerVolumeSpecName "kube-api-access-l29ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:01.264546 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.264513    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29ss\" (UniqueName: \"kubernetes.io/projected/88b8747f-33a6-4b68-9de1-e079a5ee7ac7-kube-api-access-l29ss\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:01.363034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.362943    4827 generic.go:356] "Generic (PLEG): container finished" podID="88b8747f-33a6-4b68-9de1-e079a5ee7ac7" containerID="f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4" exitCode=0
Oct 29 23:04:01.363034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.363008    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"
Oct 29 23:04:01.363034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.363021    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw" event={"ID":"88b8747f-33a6-4b68-9de1-e079a5ee7ac7","Type":"ContainerDied","Data":"f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4"}
Oct 29 23:04:01.363281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.363103    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw" event={"ID":"88b8747f-33a6-4b68-9de1-e079a5ee7ac7","Type":"ContainerDied","Data":"37acfe489729e6809ee4a0f64681a3f5e43be2c7048316985767f9eb9a32db7d"}
Oct 29 23:04:01.363281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.363127    4827 scope.go:122] "RemoveContainer" containerID="f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4"
Oct 29 23:04:01.377845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.377825    4827 scope.go:122] "RemoveContainer" containerID="f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4"
Oct 29 23:04:01.378179 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:01.378157    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4\": container with ID starting with f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4 not found: ID does not exist" containerID="f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4"
Oct 29 23:04:01.378255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.378187    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4"} err="failed to get container status \"f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4\": rpc error: code = NotFound desc = could not find container \"f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4\": container with ID starting with f6775c90e76ea178a45c147b533192f351ec7beb5e8678d8e3e932d44485adb4 not found: ID does not exist"
Oct 29 23:04:01.416290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.416262    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"]
Oct 29 23:04:01.425873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.425833    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0266dd-85bb65f5b7-bwtkw"]
Oct 29 23:04:01.730450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.730419    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"]
Oct 29 23:04:01.731142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.731126    4827 cpu_manager.go:414] "containerMap: removing container" podUID="88b8747f-33a6-4b68-9de1-e079a5ee7ac7" containerName="payload-pre-processing"
Oct 29 23:04:01.731142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.731143    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b8747f-33a6-4b68-9de1-e079a5ee7ac7" containerName="payload-pre-processing"
Oct 29 23:04:01.731292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.731282    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b8747f-33a6-4b68-9de1-e079a5ee7ac7" containerName="payload-pre-processing"
Oct 29 23:04:01.736279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.736261    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.751161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.751137    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-14debc-serving-cert\""
Oct 29 23:04:01.757508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.757485    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"]
Oct 29 23:04:01.873485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.873452    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98991aa-3271-4b69-9b1a-191af1e3a47c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.873936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.873676    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-metrics-certs\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.873936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.873713    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7nk\" (UniqueName: \"kubernetes.io/projected/f98991aa-3271-4b69-9b1a-191af1e3a47c-kube-api-access-sz7nk\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.873936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.873746    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.985936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.976023    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98991aa-3271-4b69-9b1a-191af1e3a47c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.985936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.976381    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-metrics-certs\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.985936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.976435    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7nk\" (UniqueName: \"kubernetes.io/projected/f98991aa-3271-4b69-9b1a-191af1e3a47c-kube-api-access-sz7nk\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.985936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.976474    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:01.985936 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:01.976653    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-serving-cert: secret "maas-api-e2e-praxis-a-14debc-serving-cert" not found
Oct 29 23:04:01.985936 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:01.976716    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls podName:f98991aa-3271-4b69-9b1a-191af1e3a47c nodeName:}" failed. No retries permitted until 2026-10-29 23:04:02.476696431 +0000 UTC m=+1278.099607452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls") pod "maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f" (UID: "f98991aa-3271-4b69-9b1a-191af1e3a47c") : secret "maas-api-e2e-praxis-a-14debc-serving-cert" not found
Oct 29 23:04:01.985936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:01.977675    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98991aa-3271-4b69-9b1a-191af1e3a47c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:02.019843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:02.010525    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-metrics-certs\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:02.019843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:02.019386    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7nk\" (UniqueName: \"kubernetes.io/projected/f98991aa-3271-4b69-9b1a-191af1e3a47c-kube-api-access-sz7nk\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:02.489567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:02.489537    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b8747f-33a6-4b68-9de1-e079a5ee7ac7" path="/var/lib/kubelet/pods/88b8747f-33a6-4b68-9de1-e079a5ee7ac7/volumes"
Oct 29 23:04:02.490510 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:02.490490    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:02.492969 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:02.492943    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:02.656798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:02.654423    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:03.008037 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:03.008001    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:03.021325 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:03.021284    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:03.024629 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:03.024561    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:04:03.285704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:03.285601    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:04:03.324043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:03.313576    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:04:03.423654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:03.423610    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"]
Oct 29 23:04:03.570497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:03.570225    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f" event={"ID":"f98991aa-3271-4b69-9b1a-191af1e3a47c","Type":"ContainerStarted","Data":"7bf5c4f7e2c63e579909b7774640f7a44d2d92edb30c8381b34fedcf730dc4e9"}
Oct 29 23:04:03.572301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:03.572273    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"]
Oct 29 23:04:03.573676 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:03.573626    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b" podUID="335e3b92-663b-4c2d-9efa-77bea4a7fa1d" containerName="maas-api" containerID="cri-o://8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6" gracePeriod=30
Oct 29 23:04:03.654357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:03.628920    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b" podUID="335e3b92-663b-4c2d-9efa-77bea4a7fa1d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.165:8443/health\": dial tcp 10.134.0.165:8443: connect: connection refused"
Oct 29 23:04:04.350455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.349670    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:04:04.350455 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:04.349750    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:04.408901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.408873    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:04:04.441305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.439932    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-metrics-certs\") pod \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") "
Oct 29 23:04:04.441305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.439993    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-maas-api-tls\") pod \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") "
Oct 29 23:04:04.441305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.440279    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-kube-api-access-pvrgm\" (UniqueName: \"kubernetes.io/projected/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-kube-api-access-pvrgm\") pod \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") "
Oct 29 23:04:04.441305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.440372    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-openshift-service-ca\") pod \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\" (UID: \"335e3b92-663b-4c2d-9efa-77bea4a7fa1d\") "
Oct 29 23:04:04.441914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.441889    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-openshift-service-ca" pod "335e3b92-663b-4c2d-9efa-77bea4a7fa1d" (UID: "335e3b92-663b-4c2d-9efa-77bea4a7fa1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:04.445421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.444680    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-maas-api-tls" pod "335e3b92-663b-4c2d-9efa-77bea4a7fa1d" (UID: "335e3b92-663b-4c2d-9efa-77bea4a7fa1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:04.466956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.465237    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-kube-api-access-pvrgm" pod "335e3b92-663b-4c2d-9efa-77bea4a7fa1d" (UID: "335e3b92-663b-4c2d-9efa-77bea4a7fa1d"). InnerVolumeSpecName "kube-api-access-pvrgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:04.473303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.471406    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-metrics-certs" pod "335e3b92-663b-4c2d-9efa-77bea4a7fa1d" (UID: "335e3b92-663b-4c2d-9efa-77bea4a7fa1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:04.542661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.542629    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.542661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.542661    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.542912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.542674    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrgm\" (UniqueName: \"kubernetes.io/projected/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-kube-api-access-pvrgm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.542912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.542689    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335e3b92-663b-4c2d-9efa-77bea4a7fa1d-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.597187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.597147    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f" event={"ID":"f98991aa-3271-4b69-9b1a-191af1e3a47c","Type":"ContainerStarted","Data":"1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a"}
Oct 29 23:04:04.598018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.597900    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:04.599612 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.599587    4827 generic.go:356] "Generic (PLEG): container finished" podID="335e3b92-663b-4c2d-9efa-77bea4a7fa1d" containerID="8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6" exitCode=0
Oct 29 23:04:04.599751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.599626    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b" event={"ID":"335e3b92-663b-4c2d-9efa-77bea4a7fa1d","Type":"ContainerDied","Data":"8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6"}
Oct 29 23:04:04.599751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.599632    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"
Oct 29 23:04:04.599751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.599661    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b" event={"ID":"335e3b92-663b-4c2d-9efa-77bea4a7fa1d","Type":"ContainerDied","Data":"1de4a63bc24183401297b253d5ca3c6b26e3e2fa1c06674ffa7df3646bac5321"}
Oct 29 23:04:04.599751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.599682    4827 scope.go:122] "RemoveContainer" containerID="8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6"
Oct 29 23:04:04.610003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.609982    4827 scope.go:122] "RemoveContainer" containerID="8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6"
Oct 29 23:04:04.610258 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:04.610237    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6\": container with ID starting with 8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6 not found: ID does not exist" containerID="8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6"
Oct 29 23:04:04.610323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.610263    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6"} err="failed to get container status \"8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6\": rpc error: code = NotFound desc = could not find container \"8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6\": container with ID starting with 8d111ffdf9dc43804d879c51d3a54d5354ad044998f9d3def3cfcdb5773e4de6 not found: ID does not exist"
Oct 29 23:04:04.643293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.643242    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f" podStartSLOduration=3.06120546 podStartE2EDuration="3.643229909s" totalImagesPullingTime="582.024449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:04.639950761 +0000 UTC m=+1280.262861781" watchObservedRunningTime="2026-10-29 23:04:04.643229909 +0000 UTC m=+1280.266140939"
Oct 29 23:04:04.679759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.679717    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"]
Oct 29 23:04:04.683934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:04.683909    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0266dd-5f865cb968-8sk5b"]
Oct 29 23:04:06.489066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:06.489037    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335e3b92-663b-4c2d-9efa-77bea4a7fa1d" path="/var/lib/kubelet/pods/335e3b92-663b-4c2d-9efa-77bea4a7fa1d/volumes"
Oct 29 23:04:10.349981 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:10.349936    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:10.357800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:10.357756    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:04:10.358232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:10.358192    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:04:10.614211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:10.614140    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:04:12.103242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:12.103209    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"]
Oct 29 23:04:12.103777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:12.103468    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy" containerID="cri-o://4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe" gracePeriod=30
Oct 29 23:04:12.116686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:12.116646    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"]
Oct 29 23:04:12.117367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:12.117299    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy" containerID="cri-o://7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201" gracePeriod=30
Oct 29 23:04:12.138844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:12.129664    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"]
Oct 29 23:04:12.138844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:12.129988    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy" containerID="cri-o://7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650" gracePeriod=30
Oct 29 23:04:13.913987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:13.913953    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"]
Oct 29 23:04:13.914873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:13.914847    4827 cpu_manager.go:414] "containerMap: removing container" podUID="335e3b92-663b-4c2d-9efa-77bea4a7fa1d" containerName="maas-api"
Oct 29 23:04:13.914873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:13.914872    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335e3b92-663b-4c2d-9efa-77bea4a7fa1d" containerName="maas-api"
Oct 29 23:04:13.915156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:13.915132    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="335e3b92-663b-4c2d-9efa-77bea4a7fa1d" containerName="maas-api"
Oct 29 23:04:13.920833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:13.920816    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:13.939646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:13.939625    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"]
Oct 29 23:04:13.946911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:13.946884    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c73f1d-serving-cert\""
Oct 29 23:04:14.028864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.028824    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.029045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.028962    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.029117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.029098    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtnvq\" (UniqueName: \"kubernetes.io/projected/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-kube-api-access-dtnvq\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.029196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.029178    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-metrics-certs\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.131414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.131375    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtnvq\" (UniqueName: \"kubernetes.io/projected/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-kube-api-access-dtnvq\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.131621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.131476    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-metrics-certs\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.131740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.131721    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.131856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.131840    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.131993 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:14.131844    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-serving-cert: secret "maas-api-e2e-praxis-b-c73f1d-serving-cert" not found
Oct 29 23:04:14.132116 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:14.132053    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls podName:3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe nodeName:}" failed. No retries permitted until 2026-10-29 23:04:14.632031744 +0000 UTC m=+1290.254942766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls") pod "maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" (UID: "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe") : secret "maas-api-e2e-praxis-b-c73f1d-serving-cert" not found
Oct 29 23:04:14.132472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.132450    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.134199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.134178    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-metrics-certs\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.155379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.155350    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtnvq\" (UniqueName: \"kubernetes.io/projected/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-kube-api-access-dtnvq\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.443663 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:14.443620    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:14.647029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.646996    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.649453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.649422    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:14.835196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:14.835116    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:15.238497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:15.217223    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"]
Oct 29 23:04:15.838308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:15.838273    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" event={"ID":"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe","Type":"ContainerStarted","Data":"a664fbb31931a1fe5e19008fdfa8db6290942d78e282b887e8626c84910e0c26"}
Oct 29 23:04:15.838468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:15.838363    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:15.838468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:15.838382    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" event={"ID":"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe","Type":"ContainerStarted","Data":"6701e509a9359f47bfda2a1550c5ebaeba7de10b991c48177654eb08f1362ede"}
Oct 29 23:04:15.882333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:15.882281    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" podStartSLOduration=2.419414816 podStartE2EDuration="2.882266046s" totalImagesPullingTime="462.85123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:15.879373073 +0000 UTC m=+1291.502284093" watchObservedRunningTime="2026-10-29 23:04:15.882266046 +0000 UTC m=+1291.505177066"
Oct 29 23:04:17.442686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.442653    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:04:17.482165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.482143    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:04:17.500666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.500643    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:04:17.526413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526374    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-kube-api-access-tztnh\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-kube-api-access-tztnh\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.526596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526481    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-credential-socket\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.526596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526511    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-token\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-token\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.526596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526537    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e856e7-5135-4814-beaf-7f4c10708fd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e856e7-5135-4814-beaf-7f4c10708fd3-istiod-ca-cert\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.526596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526560    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-certs\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.526596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526589    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-data\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.526903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526841    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-podinfo\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.526903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.526855    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-credential-socket" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.527025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.527001    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-socket\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.527087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.527067    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-envoy\") pod \"07e856e7-5135-4814-beaf-7f4c10708fd3\" (UID: \"07e856e7-5135-4814-beaf-7f4c10708fd3\") "
Oct 29 23:04:17.527428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.527192    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-certs" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.527428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.527332    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-data" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.527428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.527394    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-socket" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.527664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.527524    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e856e7-5135-4814-beaf-7f4c10708fd3-istiod-ca-cert" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:17.528104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.528002    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.528104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.528030    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.528104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.528046    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e856e7-5135-4814-beaf-7f4c10708fd3-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.528104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.528059    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.528104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.528070    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.529456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.529431    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-envoy" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.529712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.529694    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-kube-api-access-tztnh" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "kube-api-access-tztnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.529946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.529866    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-podinfo" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:17.531062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.531039    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-token" pod "07e856e7-5135-4814-beaf-7f4c10708fd3" (UID: "07e856e7-5135-4814-beaf-7f4c10708fd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.629384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629351    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85c2fbac-aceb-4938-9032-537bf3808f96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2fbac-aceb-4938-9032-537bf3808f96-istio-podinfo\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.629570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629403    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-podinfo\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.629570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629479    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-envoy\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.629570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629516    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-istio-token\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-istio-token\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.629812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629580    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c2fbac-aceb-4938-9032-537bf3808f96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2fbac-aceb-4938-9032-537bf3808f96-istiod-ca-cert\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.629812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629624    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-certs\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.629812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629678    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-socket\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.629812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629718    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istiod-ca-cert\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.629812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629746    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-envoy\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.630107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629829    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-token\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-token\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.630107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.629911    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-kube-api-access-l4cjm\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-kube-api-access-l4cjm\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.630107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630005    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-socket\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.630107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630009    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c2fbac-aceb-4938-9032-537bf3808f96-istiod-ca-cert" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:17.630107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630069    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-data\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.630372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630112    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-credential-socket\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.630372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630153    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-kube-api-access-hbh9t\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-kube-api-access-hbh9t\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.630372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630260    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-socket" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.630372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630277    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-certs\") pod \"85c2fbac-aceb-4938-9032-537bf3808f96\" (UID: \"85c2fbac-aceb-4938-9032-537bf3808f96\") "
Oct 29 23:04:17.630372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630316    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-data\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.630372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630358    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-credential-socket\") pod \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\" (UID: \"bb46a1dd-0383-4d4b-840e-6e45e6c76046\") "
Oct 29 23:04:17.630372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630360    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istiod-ca-cert" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:17.630788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630629    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-certs" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.631123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.630867    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-credential-socket" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.631400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631378    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631407    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2fbac-aceb-4938-9032-537bf3808f96-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631422    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631435    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631448    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tztnh\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-kube-api-access-tztnh\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631464    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631479    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631493    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.631863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.631507    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e856e7-5135-4814-beaf-7f4c10708fd3-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.632110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.632078    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-socket" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.632910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.632880    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-certs" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.633086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.633065    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-data" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.633292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.633269    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-data" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.633619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.633598    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-credential-socket" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.634742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.634715    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-token" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.634860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.634741    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-kube-api-access-hbh9t" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "kube-api-access-hbh9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.635070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.635050    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-envoy" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.636138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.636110    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-kube-api-access-l4cjm" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "kube-api-access-l4cjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.636138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.636129    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-istio-token" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.636289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.636131    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-podinfo" pod "bb46a1dd-0383-4d4b-840e-6e45e6c76046" (UID: "bb46a1dd-0383-4d4b-840e-6e45e6c76046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:17.637003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.636972    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-envoy" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.637330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.637304    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85c2fbac-aceb-4938-9032-537bf3808f96-istio-podinfo" pod "85c2fbac-aceb-4938-9032-537bf3808f96" (UID: "85c2fbac-aceb-4938-9032-537bf3808f96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:17.733079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733005    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733041    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733055    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733069    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733082    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733095    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4cjm\" (UniqueName: \"kubernetes.io/projected/bb46a1dd-0383-4d4b-840e-6e45e6c76046-kube-api-access-l4cjm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733110    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733124    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733138    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbh9t\" (UniqueName: \"kubernetes.io/projected/85c2fbac-aceb-4938-9032-537bf3808f96-kube-api-access-hbh9t\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733154    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2fbac-aceb-4938-9032-537bf3808f96-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733167    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733179    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2fbac-aceb-4938-9032-537bf3808f96-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.733321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.733193    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb46a1dd-0383-4d4b-840e-6e45e6c76046-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.855048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.855016    4827 generic.go:356] "Generic (PLEG): container finished" podID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerID="4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe" exitCode=0
Oct 29 23:04:17.855232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.855073    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"
Oct 29 23:04:17.855232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.855107    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" event={"ID":"bb46a1dd-0383-4d4b-840e-6e45e6c76046","Type":"ContainerDied","Data":"4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe"}
Oct 29 23:04:17.855232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.855165    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8" event={"ID":"bb46a1dd-0383-4d4b-840e-6e45e6c76046","Type":"ContainerDied","Data":"8cb8d0cf35adb1c949a33ac854cba82c7a4c595e9016f1c545eed7f47a0f6f36"}
Oct 29 23:04:17.855232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.855195    4827 scope.go:122] "RemoveContainer" containerID="4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe"
Oct 29 23:04:17.856896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.856868    4827 generic.go:356] "Generic (PLEG): container finished" podID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerID="7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201" exitCode=0
Oct 29 23:04:17.857019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.856919    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"
Oct 29 23:04:17.857019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.856960    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" event={"ID":"07e856e7-5135-4814-beaf-7f4c10708fd3","Type":"ContainerDied","Data":"7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201"}
Oct 29 23:04:17.857138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.857028    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs" event={"ID":"07e856e7-5135-4814-beaf-7f4c10708fd3","Type":"ContainerDied","Data":"d3d219c799b590ec2554a6eed3e95696bf50345796f02ffbe8d8bf8378ea4f8b"}
Oct 29 23:04:17.858525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.858504    4827 generic.go:356] "Generic (PLEG): container finished" podID="85c2fbac-aceb-4938-9032-537bf3808f96" containerID="7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650" exitCode=0
Oct 29 23:04:17.858648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.858625    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" event={"ID":"85c2fbac-aceb-4938-9032-537bf3808f96","Type":"ContainerDied","Data":"7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650"}
Oct 29 23:04:17.858801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.858670    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6" event={"ID":"85c2fbac-aceb-4938-9032-537bf3808f96","Type":"ContainerDied","Data":"50b7bd17722e57b688eb8cece2230f4e4073dab5896279bf8d3b2bc239a606a4"}
Oct 29 23:04:17.858801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.858691    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"
Oct 29 23:04:17.869333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.869316    4827 scope.go:122] "RemoveContainer" containerID="89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48"
Oct 29 23:04:17.879877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.879861    4827 scope.go:122] "RemoveContainer" containerID="4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe"
Oct 29 23:04:17.880147 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:17.880128    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe\": container with ID starting with 4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe not found: ID does not exist" containerID="4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe"
Oct 29 23:04:17.880223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.880159    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe"} err="failed to get container status \"4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe\": rpc error: code = NotFound desc = could not find container \"4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe\": container with ID starting with 4d9cf61dbdcf5a0d9cc3fdbc59c09b2c6fae3fa70393617471319a1a8957d3fe not found: ID does not exist"
Oct 29 23:04:17.880223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.880184    4827 scope.go:122] "RemoveContainer" containerID="89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48"
Oct 29 23:04:17.880448 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:17.880431    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48\": container with ID starting with 89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48 not found: ID does not exist" containerID="89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48"
Oct 29 23:04:17.880489 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.880454    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48"} err="failed to get container status \"89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48\": rpc error: code = NotFound desc = could not find container \"89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48\": container with ID starting with 89d99ed1004b72f1f25455ce8e171258621d90f1e18dcad7431d7af70f32cc48 not found: ID does not exist"
Oct 29 23:04:17.880489 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.880470    4827 scope.go:122] "RemoveContainer" containerID="7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201"
Oct 29 23:04:17.890064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.890044    4827 scope.go:122] "RemoveContainer" containerID="7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201"
Oct 29 23:04:17.890292 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:17.890275    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201\": container with ID starting with 7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201 not found: ID does not exist" containerID="7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201"
Oct 29 23:04:17.890352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.890297    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201"} err="failed to get container status \"7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201\": rpc error: code = NotFound desc = could not find container \"7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201\": container with ID starting with 7f946cbb6f35af5504a86ead29fd52f4805580499c168abe52d9bcc86e57a201 not found: ID does not exist"
Oct 29 23:04:17.890352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.890311    4827 scope.go:122] "RemoveContainer" containerID="7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650"
Oct 29 23:04:17.902410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.902378    4827 scope.go:122] "RemoveContainer" containerID="7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650"
Oct 29 23:04:17.902638 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:17.902622    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650\": container with ID starting with 7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650 not found: ID does not exist" containerID="7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650"
Oct 29 23:04:17.902699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.902643    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650"} err="failed to get container status \"7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650\": rpc error: code = NotFound desc = could not find container \"7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650\": container with ID starting with 7f371196f3bf41119e0770ccf220da3a7bd8c03cd35cf69c61df09ab5d15b650 not found: ID does not exist"
Oct 29 23:04:17.908748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.908726    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"]
Oct 29 23:04:17.920602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.920581    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gr5x8"]
Oct 29 23:04:17.985439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.985336    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"]
Oct 29 23:04:17.993477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:17.993445    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-zfnqs"]
Oct 29 23:04:18.032957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:18.032930    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"]
Oct 29 23:04:18.037070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:18.037041    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-wcdc6"]
Oct 29 23:04:18.490931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:18.490900    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" path="/var/lib/kubelet/pods/07e856e7-5135-4814-beaf-7f4c10708fd3/volumes"
Oct 29 23:04:18.491398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:18.491383    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" path="/var/lib/kubelet/pods/85c2fbac-aceb-4938-9032-537bf3808f96/volumes"
Oct 29 23:04:18.491894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:18.491879    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" path="/var/lib/kubelet/pods/bb46a1dd-0383-4d4b-840e-6e45e6c76046/volumes"
Oct 29 23:04:19.939796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.938579    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"]
Oct 29 23:04:19.939796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939760    4827 cpu_manager.go:414] "containerMap: removing container" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy"
Oct 29 23:04:19.939796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939798    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939858    4827 cpu_manager.go:414] "containerMap: removing container" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939866    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939900    4827 cpu_manager.go:414] "containerMap: removing container" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939908    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939923    4827 cpu_manager.go:414] "containerMap: removing container" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.939930    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.940114    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c2fbac-aceb-4938-9032-537bf3808f96" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.940128    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy"
Oct 29 23:04:19.940412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:19.940147    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e856e7-5135-4814-beaf-7f4c10708fd3" containerName="istio-proxy"
Oct 29 23:04:20.109645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.108218    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"]
Oct 29 23:04:20.109645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.108362    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.112023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.111991    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd352a-1d5d-473d-9e13-930a469ad580-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.112154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.112071    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvq4\" (UniqueName: \"kubernetes.io/projected/9afd352a-1d5d-473d-9e13-930a469ad580-kube-api-access-xzvq4\") pod \"payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.136033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.136009    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c88955-dockercfg-lw6zp\""
Oct 29 23:04:20.136172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.136066    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c88955\""
Oct 29 23:04:20.213605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.213515    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd352a-1d5d-473d-9e13-930a469ad580-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.213605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.213566    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvq4\" (UniqueName: \"kubernetes.io/projected/9afd352a-1d5d-473d-9e13-930a469ad580-kube-api-access-xzvq4\") pod \"payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.214193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.214176    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd352a-1d5d-473d-9e13-930a469ad580-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.237230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.237200    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvq4\" (UniqueName: \"kubernetes.io/projected/9afd352a-1d5d-473d-9e13-930a469ad580-kube-api-access-xzvq4\") pod \"payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.429297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.429259    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:20.744070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.744046    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"]
Oct 29 23:04:20.744842 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:04:20.744812    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afd352a_1d5d_473d_9e13_930a469ad580.slice/crio-d49cd76708e82f0bd80c23e372ef31a768c703c0fc7393f525a7e166353e975d WatchSource:0}: Error finding container d49cd76708e82f0bd80c23e372ef31a768c703c0fc7393f525a7e166353e975d: Status 404 returned error can't find the container with id d49cd76708e82f0bd80c23e372ef31a768c703c0fc7393f525a7e166353e975d
Oct 29 23:04:20.957531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.957492    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt" event={"ID":"9afd352a-1d5d-473d-9e13-930a469ad580","Type":"ContainerStarted","Data":"3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954"}
Oct 29 23:04:20.958063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.957600    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt" event={"ID":"9afd352a-1d5d-473d-9e13-930a469ad580","Type":"ContainerStarted","Data":"d49cd76708e82f0bd80c23e372ef31a768c703c0fc7393f525a7e166353e975d"}
Oct 29 23:04:20.958063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:20.957821    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:21.002426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:21.002334    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt" podStartSLOduration=2.002319615 podStartE2EDuration="2.002319615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:20.998014379 +0000 UTC m=+1296.620925418" watchObservedRunningTime="2026-10-29 23:04:21.002319615 +0000 UTC m=+1296.625230635"
Oct 29 23:04:21.852957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:21.852931    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:04:24.519656 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:24.519622    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:25.190408 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:25.190370    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:27.381849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.381762    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"]
Oct 29 23:04:27.382599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.382583    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb46a1dd-0383-4d4b-840e-6e45e6c76046" containerName="istio-proxy"
Oct 29 23:04:27.387245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.387229    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.434590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.427035    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-182195-serving-cert\""
Oct 29 23:04:27.441444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.440796    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"]
Oct 29 23:04:27.465840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.465812    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-metrics-certs\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.466014 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.465929    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96jtv\" (UniqueName: \"kubernetes.io/projected/36b8ec37-28c8-4e00-8810-49379546e146-kube-api-access-96jtv\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.466102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.466022    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-maas-api-tls\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.466102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.466054    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b8ec37-28c8-4e00-8810-49379546e146-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.577785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.575871    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-metrics-certs\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.577785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.576067    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96jtv\" (UniqueName: \"kubernetes.io/projected/36b8ec37-28c8-4e00-8810-49379546e146-kube-api-access-96jtv\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.577785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.576208    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-maas-api-tls\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.577785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.576250    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b8ec37-28c8-4e00-8810-49379546e146-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.591789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.584196    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b8ec37-28c8-4e00-8810-49379546e146-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.633835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.631396    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-maas-api-tls\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.633835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.632037    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-metrics-certs\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.641786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.641513    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96jtv\" (UniqueName: \"kubernetes.io/projected/36b8ec37-28c8-4e00-8810-49379546e146-kube-api-access-96jtv\") pod \"maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:27.707615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:27.707574    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:28.015066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:28.015043    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"]
Oct 29 23:04:28.017214 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:04:28.017187    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b8ec37_28c8_4e00_8810_49379546e146.slice/crio-b0fcb95e716953842c6e82c6e4d0a0ad786e93533cb828ddd02f0fe5d40c1c6e WatchSource:0}: Error finding container b0fcb95e716953842c6e82c6e4d0a0ad786e93533cb828ddd02f0fe5d40c1c6e: Status 404 returned error can't find the container with id b0fcb95e716953842c6e82c6e4d0a0ad786e93533cb828ddd02f0fe5d40c1c6e
Oct 29 23:04:28.043039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:28.043012    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z" event={"ID":"36b8ec37-28c8-4e00-8810-49379546e146","Type":"ContainerStarted","Data":"b0fcb95e716953842c6e82c6e4d0a0ad786e93533cb828ddd02f0fe5d40c1c6e"}
Oct 29 23:04:29.071093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:29.071046    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z" event={"ID":"36b8ec37-28c8-4e00-8810-49379546e146","Type":"ContainerStarted","Data":"6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69"}
Oct 29 23:04:29.071600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:29.071413    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:29.132328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:29.131923    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z" podStartSLOduration=1.713269355 podStartE2EDuration="2.131903208s" totalImagesPullingTime="418.633853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:29.125745529 +0000 UTC m=+1304.748656579" watchObservedRunningTime="2026-10-29 23:04:29.131903208 +0000 UTC m=+1304.754814231"
Oct 29 23:04:31.968346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:31.968316    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:04:32.825072 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:32.825036    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:04:34.596096 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:34.596058    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:35.084792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:35.084752    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:04:40.252788 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:40.252725    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad67055_0ab0_4997_b160_499db019ba1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice/crio-3b79d474597855ca00e1e343af2e79d58be4b19d13b57f34fcd70943ee10564a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e72a9_9c4c_4522_81bc_479aac358efe.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:04:41.580958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.580926    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"]
Oct 29 23:04:41.586520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.586496    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.607784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.605277    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4f2e3a-openshift-default-dockercfg-krjrl\""
Oct 29 23:04:41.621782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.621674    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"]
Oct 29 23:04:41.633616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.633588    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-certs\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.633746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.633633    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-data\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.633746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.633651    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-socket\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.633895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.633870    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-envoy\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.633975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.633899    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f062884e-7973-4797-854e-d2322a868e86-istiod-ca-cert\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.634019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.633987    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-istio-token\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.634126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.634087    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-credential-socket\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.634254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.634240    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f062884e-7973-4797-854e-d2322a868e86-istio-podinfo\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.634345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.634326    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwrp\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-kube-api-access-plwrp\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.735537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735502    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-certs\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.735716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735550    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-data\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.735716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735566    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-socket\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.735716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735621    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-envoy\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.735716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735639    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f062884e-7973-4797-854e-d2322a868e86-istiod-ca-cert\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.735716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735695    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-istio-token\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735744    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-credential-socket\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735868    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f062884e-7973-4797-854e-d2322a868e86-istio-podinfo\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.735950    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwrp\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-kube-api-access-plwrp\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.736028    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-data\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.736049    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-socket\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.736438    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-certs\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.736622    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-credential-socket\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.736916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.736895    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f062884e-7973-4797-854e-d2322a868e86-istiod-ca-cert\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.738161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.738142    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-envoy\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.738621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.738605    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f062884e-7973-4797-854e-d2322a868e86-istio-podinfo\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.759164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.759136    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-istio-token\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.759288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.759271    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwrp\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-kube-api-access-plwrp\") pod \"e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") " pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:41.922375 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:41.922075    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:42.393839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:42.393809    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"]
Oct 29 23:04:42.394166 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:04:42.394141    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf062884e_7973_4797_854e_d2322a868e86.slice/crio-b0dff3be1f2ac19a6a8f3a5fdf002afd8144485aa5123d0a3d3366c750f29fa1 WatchSource:0}: Error finding container b0dff3be1f2ac19a6a8f3a5fdf002afd8144485aa5123d0a3d3366c750f29fa1: Status 404 returned error can't find the container with id b0dff3be1f2ac19a6a8f3a5fdf002afd8144485aa5123d0a3d3366c750f29fa1
Oct 29 23:04:42.396909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:42.396876    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:42.397020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:42.396942    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:42.397020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:42.396971    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:42.921322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:42.921284    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"]
Oct 29 23:04:42.921810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:42.921540    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv" podUID="5f772756-617a-4aef-baae-b56289fd1de2" containerName="maas-api" containerID="cri-o://5e8723713fb78ccf62978ee8ebb7644b957bb7bcafed6ff2fc087e2b5a2430d6" gracePeriod=30
Oct 29 23:04:43.208839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.196094    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" event={"ID":"f062884e-7973-4797-854e-d2322a868e86","Type":"ContainerStarted","Data":"299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a"}
Oct 29 23:04:43.208839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.196159    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" event={"ID":"f062884e-7973-4797-854e-d2322a868e86","Type":"ContainerStarted","Data":"b0dff3be1f2ac19a6a8f3a5fdf002afd8144485aa5123d0a3d3366c750f29fa1"}
Oct 29 23:04:43.208839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.198674    4827 generic.go:356] "Generic (PLEG): container finished" podID="5f772756-617a-4aef-baae-b56289fd1de2" containerID="5e8723713fb78ccf62978ee8ebb7644b957bb7bcafed6ff2fc087e2b5a2430d6" exitCode=0
Oct 29 23:04:43.208839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.199030    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv" event={"ID":"5f772756-617a-4aef-baae-b56289fd1de2","Type":"ContainerDied","Data":"5e8723713fb78ccf62978ee8ebb7644b957bb7bcafed6ff2fc087e2b5a2430d6"}
Oct 29 23:04:43.296965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.295269    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podStartSLOduration=2.295247329 podStartE2EDuration="2.295247329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:43.256360139 +0000 UTC m=+1318.879271172" watchObservedRunningTime="2026-10-29 23:04:43.295247329 +0000 UTC m=+1318.918158350"
Oct 29 23:04:43.329790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.323725    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"]
Oct 29 23:04:43.329790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.324459    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2" podUID="bd9c0a0d-739e-48db-80cd-739869e11aae" containerName="maas-api" containerID="cri-o://c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75" gracePeriod=30
Oct 29 23:04:43.544593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.544561    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:04:43.648867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.648839    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls\") pod \"5f772756-617a-4aef-baae-b56289fd1de2\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") "
Oct 29 23:04:43.649051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.648974    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f772756-617a-4aef-baae-b56289fd1de2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f772756-617a-4aef-baae-b56289fd1de2-openshift-service-ca\") pod \"5f772756-617a-4aef-baae-b56289fd1de2\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") "
Oct 29 23:04:43.649125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.649068    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f772756-617a-4aef-baae-b56289fd1de2-kube-api-access-zpp45\" (UniqueName: \"kubernetes.io/projected/5f772756-617a-4aef-baae-b56289fd1de2-kube-api-access-zpp45\") pod \"5f772756-617a-4aef-baae-b56289fd1de2\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") "
Oct 29 23:04:43.649125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.649098    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-metrics-certs\") pod \"5f772756-617a-4aef-baae-b56289fd1de2\" (UID: \"5f772756-617a-4aef-baae-b56289fd1de2\") "
Oct 29 23:04:43.649887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.649845    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f772756-617a-4aef-baae-b56289fd1de2-openshift-service-ca" pod "5f772756-617a-4aef-baae-b56289fd1de2" (UID: "5f772756-617a-4aef-baae-b56289fd1de2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:43.652013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.651987    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls" pod "5f772756-617a-4aef-baae-b56289fd1de2" (UID: "5f772756-617a-4aef-baae-b56289fd1de2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:43.652259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.652235    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-metrics-certs" pod "5f772756-617a-4aef-baae-b56289fd1de2" (UID: "5f772756-617a-4aef-baae-b56289fd1de2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:43.653104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.653081    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f772756-617a-4aef-baae-b56289fd1de2-kube-api-access-zpp45" pod "5f772756-617a-4aef-baae-b56289fd1de2" (UID: "5f772756-617a-4aef-baae-b56289fd1de2"). InnerVolumeSpecName "kube-api-access-zpp45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:43.750975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.750950    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f772756-617a-4aef-baae-b56289fd1de2-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:43.750975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.750976    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpp45\" (UniqueName: \"kubernetes.io/projected/5f772756-617a-4aef-baae-b56289fd1de2-kube-api-access-zpp45\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:43.751117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.750985    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:43.751117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.750995    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f772756-617a-4aef-baae-b56289fd1de2-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:43.788962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.788928    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:04:43.888779 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.888728    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9c0a0d-739e-48db-80cd-739869e11aae-kube-api-access-pkfzv\" (UniqueName: \"kubernetes.io/projected/bd9c0a0d-739e-48db-80cd-739869e11aae-kube-api-access-pkfzv\") pod \"bd9c0a0d-739e-48db-80cd-739869e11aae\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") "
Oct 29 23:04:43.889128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.889102    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls\") pod \"bd9c0a0d-739e-48db-80cd-739869e11aae\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") "
Oct 29 23:04:43.889560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.889540    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-metrics-certs\") pod \"bd9c0a0d-739e-48db-80cd-739869e11aae\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") "
Oct 29 23:04:43.889714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.889697    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9c0a0d-739e-48db-80cd-739869e11aae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c0a0d-739e-48db-80cd-739869e11aae-openshift-service-ca\") pod \"bd9c0a0d-739e-48db-80cd-739869e11aae\" (UID: \"bd9c0a0d-739e-48db-80cd-739869e11aae\") "
Oct 29 23:04:43.931917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.922699    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9c0a0d-739e-48db-80cd-739869e11aae-kube-api-access-pkfzv" pod "bd9c0a0d-739e-48db-80cd-739869e11aae" (UID: "bd9c0a0d-739e-48db-80cd-739869e11aae"). InnerVolumeSpecName "kube-api-access-pkfzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:43.931917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.928069    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:43.983130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.970010    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9c0a0d-739e-48db-80cd-739869e11aae-openshift-service-ca" pod "bd9c0a0d-739e-48db-80cd-739869e11aae" (UID: "bd9c0a0d-739e-48db-80cd-739869e11aae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:44.007919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:43.986158    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:44.030831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.030743    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c0a0d-739e-48db-80cd-739869e11aae-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:44.030831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.030800    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkfzv\" (UniqueName: \"kubernetes.io/projected/bd9c0a0d-739e-48db-80cd-739869e11aae-kube-api-access-pkfzv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:44.066508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.059091    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls" pod "bd9c0a0d-739e-48db-80cd-739869e11aae" (UID: "bd9c0a0d-739e-48db-80cd-739869e11aae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:44.086073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.085935    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-metrics-certs" pod "bd9c0a0d-739e-48db-80cd-739869e11aae" (UID: "bd9c0a0d-739e-48db-80cd-739869e11aae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:44.165821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.162719    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:44.165821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.162752    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c0a0d-739e-48db-80cd-739869e11aae-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:44.231113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.231073    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv" event={"ID":"5f772756-617a-4aef-baae-b56289fd1de2","Type":"ContainerDied","Data":"deeff343b6f3785c0d996c44adc10a2349ce11bad89b177526172d07661ff508"}
Oct 29 23:04:44.231369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.231346    4827 scope.go:122] "RemoveContainer" containerID="5e8723713fb78ccf62978ee8ebb7644b957bb7bcafed6ff2fc087e2b5a2430d6"
Oct 29 23:04:44.231674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.231656    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"
Oct 29 23:04:44.241445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.241414    4827 generic.go:356] "Generic (PLEG): container finished" podID="bd9c0a0d-739e-48db-80cd-739869e11aae" containerID="c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75" exitCode=0
Oct 29 23:04:44.242303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.242286    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"
Oct 29 23:04:44.242468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.242423    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2" event={"ID":"bd9c0a0d-739e-48db-80cd-739869e11aae","Type":"ContainerDied","Data":"c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75"}
Oct 29 23:04:44.242562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.242469    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2" event={"ID":"bd9c0a0d-739e-48db-80cd-739869e11aae","Type":"ContainerDied","Data":"51860718bb6e7918344d5808098d143d2d774090c33b42d3bab7153349d844f2"}
Oct 29 23:04:44.243044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.242910    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:44.244330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.244306    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:04:44.260810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.260783    4827 scope.go:122] "RemoveContainer" containerID="c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75"
Oct 29 23:04:44.275148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.275128    4827 scope.go:122] "RemoveContainer" containerID="c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75"
Oct 29 23:04:44.275668 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:44.275628    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75\": container with ID starting with c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75 not found: ID does not exist" containerID="c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75"
Oct 29 23:04:44.275782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.275688    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75"} err="failed to get container status \"c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75\": rpc error: code = NotFound desc = could not find container \"c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75\": container with ID starting with c490201cd8ebfdda7936758f436c23d3a14562292750fffa0617b15da5219f75 not found: ID does not exist"
Oct 29 23:04:44.291915 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.291881    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"]
Oct 29 23:04:44.297375 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.297350    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e1c46-79ccfcb49c-vzhtv"]
Oct 29 23:04:44.394912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.393574    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"]
Oct 29 23:04:44.404682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.404646    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-848fb8-c9dbb69cf-w4sm2"]
Oct 29 23:04:44.493358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.493312    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f772756-617a-4aef-baae-b56289fd1de2" path="/var/lib/kubelet/pods/5f772756-617a-4aef-baae-b56289fd1de2/volumes"
Oct 29 23:04:44.494184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:44.494159    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9c0a0d-739e-48db-80cd-739869e11aae" path="/var/lib/kubelet/pods/bd9c0a0d-739e-48db-80cd-739869e11aae/volumes"
Oct 29 23:04:48.078260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:48.078181    4827 scope.go:122] "RemoveContainer" containerID="4c669f16cc029e44fbff08a96ef365b629bce409852973990146b3a0808e0410"
Oct 29 23:04:48.115885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:48.115828    4827 scope.go:122] "RemoveContainer" containerID="a517c5048c96d72bd2d566f6c04dedeec2a792bc97aa40596a27161e03b95b50"
Oct 29 23:04:48.131165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:48.131140    4827 scope.go:122] "RemoveContainer" containerID="2579d31cf059e53b2eac198e63e49f85d64ec21583bf9882e8eb6617849d0659"
Oct 29 23:04:48.143296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:48.143275    4827 scope.go:122] "RemoveContainer" containerID="15ed3cff21850bffc7e9940b440df72cc9e16ba3108ece7ac769e4c6ddce01c7"
Oct 29 23:04:51.824792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:51.815352    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"]
Oct 29 23:04:51.824792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:51.815625    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy" containerID="cri-o://2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590" gracePeriod=30
Oct 29 23:04:54.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.033585    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"]
Oct 29 23:04:54.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.034820    4827 cpu_manager.go:414] "containerMap: removing container" podUID="bd9c0a0d-739e-48db-80cd-739869e11aae" containerName="maas-api"
Oct 29 23:04:54.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.034843    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9c0a0d-739e-48db-80cd-739869e11aae" containerName="maas-api"
Oct 29 23:04:54.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.034955    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5f772756-617a-4aef-baae-b56289fd1de2" containerName="maas-api"
Oct 29 23:04:54.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.034963    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f772756-617a-4aef-baae-b56289fd1de2" containerName="maas-api"
Oct 29 23:04:54.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.035123    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f772756-617a-4aef-baae-b56289fd1de2" containerName="maas-api"
Oct 29 23:04:54.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.035142    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9c0a0d-739e-48db-80cd-739869e11aae" containerName="maas-api"
Oct 29 23:04:54.081790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.068360    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"]
Oct 29 23:04:54.081790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.068489    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.095914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.095596    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3f7d4b-openshift-default-dockercfg-jrq9r\""
Oct 29 23:04:54.165791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.165676    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdc6\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-kube-api-access-5wdc6\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.165791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.165759    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-envoy\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.166029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.165826    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-socket\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.166029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.165852    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-podinfo\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.166029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.165894    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4230c6-f355-462c-8a8f-eb917819bd34-istiod-ca-cert\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.166199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.166051    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-token\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.166199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.166104    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-credential-socket\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.166199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.166129    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-certs\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.166199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.166171    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-data\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.337866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.317454    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdc6\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-kube-api-access-5wdc6\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.338415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.338377    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-envoy\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.368792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.342840    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-socket\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.369106 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.369074    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-podinfo\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.397933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.397889    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4230c6-f355-462c-8a8f-eb917819bd34-istiod-ca-cert\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.398260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.398234    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-token\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.398449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.398432    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-credential-socket\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.398587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.398571    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-certs\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.402935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.402906    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-data\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.431605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.370469    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-socket\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.431883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.411907    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4230c6-f355-462c-8a8f-eb917819bd34-istiod-ca-cert\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.432000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.412137    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-credential-socket\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.432095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.412339    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-certs\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.432178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.429164    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-data\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.538480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.536832    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-podinfo\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.669921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.665391    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:04:54.700636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.700568    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdc6\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-kube-api-access-5wdc6\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.701157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.701133    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-envoy\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.715347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.715288    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-token\") pod \"e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") " pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:54.796149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:54.796102    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:55.171717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.171689    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"]
Oct 29 23:04:55.174213 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:04:55.174180    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4230c6_f355_462c_8a8f_eb917819bd34.slice/crio-d045f1395b50d68a52ed75a8b4bea27408f4be058459dce893bed1365d208cd7 WatchSource:0}: Error finding container d045f1395b50d68a52ed75a8b4bea27408f4be058459dce893bed1365d208cd7: Status 404 returned error can't find the container with id d045f1395b50d68a52ed75a8b4bea27408f4be058459dce893bed1365d208cd7
Oct 29 23:04:55.177677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.177641    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:55.177777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.177710    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:55.177824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.177760    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:55.708192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.708152    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" event={"ID":"4b4230c6-f355-462c-8a8f-eb917819bd34","Type":"ContainerStarted","Data":"b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d"}
Oct 29 23:04:55.708345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.708214    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" event={"ID":"4b4230c6-f355-462c-8a8f-eb917819bd34","Type":"ContainerStarted","Data":"d045f1395b50d68a52ed75a8b4bea27408f4be058459dce893bed1365d208cd7"}
Oct 29 23:04:55.768525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.768461    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podStartSLOduration=2.768440286 podStartE2EDuration="2.768440286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:55.763300701 +0000 UTC m=+1331.386211721" watchObservedRunningTime="2026-10-29 23:04:55.768440286 +0000 UTC m=+1331.391351308"
Oct 29 23:04:55.797144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:55.797115    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:56.804950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:56.804917    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:57.174382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.174356    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:04:57.328947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.328902    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:04:57.329116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.328969    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:04:57.349301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349270    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-envoy\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.349441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349376    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-kube-api-access-5qzmm\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-kube-api-access-5qzmm\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.349441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349428    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-credential-socket\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.349625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349604    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-certs\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.349705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349648    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-socket\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.349814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349796    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-data\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.349893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349875    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istiod-ca-cert\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.349960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.349940    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-podinfo\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.350020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.350011    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-token\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-token\") pod \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\" (UID: \"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d\") "
Oct 29 23:04:57.351226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.350166    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-certs" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:57.351226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.350454    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-credential-socket" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:57.351226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.350574    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-socket" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:57.351226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.351134    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.351226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.351158    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.351226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.351172    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.352354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.351956    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istiod-ca-cert" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:57.352572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.352516    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-data" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:57.352668 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.352618    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-envoy" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:57.365502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.365465    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-podinfo" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:57.366964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.366894    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-kube-api-access-5qzmm" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "kube-api-access-5qzmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:57.367084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.366957    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-token" pod "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" (UID: "f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:57.453173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.453092    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzmm\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-kube-api-access-5qzmm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.453173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.453126    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.453173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.453136    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.453173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.453148    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.453173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.453161    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.453173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.453169    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.729036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.728939    4827 generic.go:356] "Generic (PLEG): container finished" podID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerID="2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590" exitCode=0
Oct 29 23:04:57.729212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.729027    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" event={"ID":"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d","Type":"ContainerDied","Data":"2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590"}
Oct 29 23:04:57.729212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.729101    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr" event={"ID":"f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d","Type":"ContainerDied","Data":"0930fac91c509c48c69935c7cdb4e0ebb02ad309f9cd7c697680ceac55417335"}
Oct 29 23:04:57.729212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.729136    4827 scope.go:122] "RemoveContainer" containerID="2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590"
Oct 29 23:04:57.729562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.729532    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:57.729562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.729617    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"
Oct 29 23:04:57.730548 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.730531    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:04:57.742955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.742933    4827 scope.go:122] "RemoveContainer" containerID="8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604"
Oct 29 23:04:57.759353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.759107    4827 scope.go:122] "RemoveContainer" containerID="2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590"
Oct 29 23:04:57.759958 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:57.759823    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590\": container with ID starting with 2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590 not found: ID does not exist" containerID="2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590"
Oct 29 23:04:57.759958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.759861    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590"} err="failed to get container status \"2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590\": rpc error: code = NotFound desc = could not find container \"2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590\": container with ID starting with 2a8870b2df6f41ebaf7eae376c4514ec789598f20e82dd8ac0471a7a3711c590 not found: ID does not exist"
Oct 29 23:04:57.759958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.759888    4827 scope.go:122] "RemoveContainer" containerID="8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604"
Oct 29 23:04:57.760227 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:04:57.760200    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604\": container with ID starting with 8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604 not found: ID does not exist" containerID="8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604"
Oct 29 23:04:57.760308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.760237    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604"} err="failed to get container status \"8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604\": rpc error: code = NotFound desc = could not find container \"8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604\": container with ID starting with 8d7ec4b4dc2eb924b64a50c481d012cf0140afcdd60e634af72f55b90ff38604 not found: ID does not exist"
Oct 29 23:04:57.838705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.838672    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"]
Oct 29 23:04:57.841470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.841442    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy"
Oct 29 23:04:57.842007 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.841988    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy"
Oct 29 23:04:57.842138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.842128    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy"
Oct 29 23:04:57.842227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.842216    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy"
Oct 29 23:04:57.842657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.842640    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy"
Oct 29 23:04:57.847355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.847310    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" containerName="istio-proxy"
Oct 29 23:04:57.854868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.854839    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"]
Oct 29 23:04:57.855080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.855047    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.862160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.862082    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-848fb8-openshift-default-7568745c57-6xfrr"]
Oct 29 23:04:57.868233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.868202    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"]
Oct 29 23:04:57.877252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.875242    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1b7950-openshift-default-dockercfg-pqvgz\""
Oct 29 23:04:57.965668 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.965630    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.966067 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.966039    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.966270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.966251    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.966407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.966392    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rgr\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-kube-api-access-h5rgr\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.966578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.966563    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.966755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.966737    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.966894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.966877    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.967041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.967024    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:57.967172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:57.967158    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.068499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.068392    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.068675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.068540    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.068675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.068604    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rgr\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-kube-api-access-h5rgr\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.068886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.068685    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.068886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.068813    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069012    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069042    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069092    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069130    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069166    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069216    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069302    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069660    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.069970 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.069952    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.072176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.072144    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.072302 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.072285    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.092684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.092658    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.093041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.093011    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rgr\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-kube-api-access-h5rgr\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.196296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.196265    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:04:58.299066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.299026    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"]
Oct 29 23:04:58.326432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.326346    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"]
Oct 29 23:04:58.491710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.491676    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d" path="/var/lib/kubelet/pods/f9e8c9af-5148-4b43-85ea-8aa3c4f81d1d/volumes"
Oct 29 23:04:58.533880 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:04:58.533841    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bcc78f_48b4_4f8a_a9ad_2c751dccb63b.slice/crio-32a48313b5abf91eab479a52a8bd37b542292f0b1d025e3de985cbdfba8d3fc5 WatchSource:0}: Error finding container 32a48313b5abf91eab479a52a8bd37b542292f0b1d025e3de985cbdfba8d3fc5: Status 404 returned error can't find the container with id 32a48313b5abf91eab479a52a8bd37b542292f0b1d025e3de985cbdfba8d3fc5
Oct 29 23:04:58.569812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.568742    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"]
Oct 29 23:04:58.608788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.599406    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:58.608788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.599513    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:58.608788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.599569    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:58.807034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:58.806982    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" event={"ID":"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b","Type":"ContainerStarted","Data":"32a48313b5abf91eab479a52a8bd37b542292f0b1d025e3de985cbdfba8d3fc5"}
Oct 29 23:04:59.824002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.823239    4827 generic.go:356] "Generic (PLEG): container finished" podID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerID="7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca" exitCode=0
Oct 29 23:04:59.824002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.823385    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" event={"ID":"c98f78c6-6603-4c29-9b23-721bacf6d8d8","Type":"ContainerDied","Data":"7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca"}
Oct 29 23:04:59.825626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.824241    4827 scope.go:122] "RemoveContainer" containerID="7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca"
Oct 29 23:04:59.826530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.826489    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:59.826779 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.826594    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:59.826779 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.826649    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:04:59.829030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.828967    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" event={"ID":"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b","Type":"ContainerStarted","Data":"66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d"}
Oct 29 23:04:59.941990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:04:59.941896    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podStartSLOduration=2.941879357 podStartE2EDuration="2.941879357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:59.936206316 +0000 UTC m=+1335.559117337" watchObservedRunningTime="2026-10-29 23:04:59.941879357 +0000 UTC m=+1335.564790370"
Oct 29 23:05:00.216681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.216622    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:05:00.242971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.242925    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 23:05:00.243293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.243266    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 23:05:00.309860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.260196    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"]
Oct 29 23:05:00.341789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.337023    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"]
Oct 29 23:05:00.341789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.337158    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:00.392790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.354897    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3f7d4b\""
Oct 29 23:05:00.392790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.355053    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3f7d4b-dockercfg-mt55q\""
Oct 29 23:05:00.515751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.515607    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b78c\" (UniqueName: \"kubernetes.io/projected/9d12ed58-e87f-4c42-9cdb-5c683354197f-kube-api-access-2b78c\") pod \"payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:00.537243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.537210    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d12ed58-e87f-4c42-9cdb-5c683354197f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:00.647261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.645325    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b78c\" (UniqueName: \"kubernetes.io/projected/9d12ed58-e87f-4c42-9cdb-5c683354197f-kube-api-access-2b78c\") pod \"payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:00.647261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.645575    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d12ed58-e87f-4c42-9cdb-5c683354197f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:00.647261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.646560    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d12ed58-e87f-4c42-9cdb-5c683354197f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:00.699887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.699845    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b78c\" (UniqueName: \"kubernetes.io/projected/9d12ed58-e87f-4c42-9cdb-5c683354197f-kube-api-access-2b78c\") pod \"payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:00.840295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.840200    4827 generic.go:356] "Generic (PLEG): container finished" podID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerID="75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda" exitCode=0
Oct 29 23:05:00.841575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.840288    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" event={"ID":"ad5dacfa-5542-4a7d-a809-f02354a36829","Type":"ContainerDied","Data":"75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda"}
Oct 29 23:05:00.841575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.840969    4827 scope.go:122] "RemoveContainer" containerID="75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda"
Oct 29 23:05:00.843266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.842957    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" event={"ID":"c98f78c6-6603-4c29-9b23-721bacf6d8d8","Type":"ContainerStarted","Data":"6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e"}
Oct 29 23:05:00.843447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.843413    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:00.843530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.843510    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:00.843590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.843562    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:00.890145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:00.890115    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:01.226996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.226873    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 23:05:01.226996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.226941    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 23:05:01.507021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.506989    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:05:01.518788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.507948    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:05:01.519090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.519056    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:05:01.522419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.522385    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"]
Oct 29 23:05:01.855062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.854976    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" event={"ID":"ad5dacfa-5542-4a7d-a809-f02354a36829","Type":"ContainerStarted","Data":"6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667"}
Oct 29 23:05:01.857359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.857262    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv" event={"ID":"9d12ed58-e87f-4c42-9cdb-5c683354197f","Type":"ContainerStarted","Data":"0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f"}
Oct 29 23:05:01.857359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.857324    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv" event={"ID":"9d12ed58-e87f-4c42-9cdb-5c683354197f","Type":"ContainerStarted","Data":"857785eeb702dbc502440a0c4090442163faa69f4c9f118a8e2e9bffcd2d099a"}
Oct 29 23:05:01.857700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.857682    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:01.961880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:01.961830    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv" podStartSLOduration=1.961815898 podStartE2EDuration="1.961815898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:01.959434177 +0000 UTC m=+1337.582345198" watchObservedRunningTime="2026-10-29 23:05:01.961815898 +0000 UTC m=+1337.584726928"
Oct 29 23:05:02.202223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:02.202187    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 23:05:02.202526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:02.202498    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 23:05:02.371791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:02.370141    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:05:02.372659 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:02.372210    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:05:02.372659 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:02.372277    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:05:02.509946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:02.506550    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:05:02.509946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:02.506607    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:05:03.212791 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:05:03.201078    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:05:03.212791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.201993    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 23:05:03.212791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.202043    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 23:05:03.337616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.316648    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm" podUID="333138a6-55c4-459d-89d2-f6044d08a481" containerName="payload-processing" containerID="cri-o://9fa0da9b859d641517e405bfc1cc5db650b44fb2f63c6f26ccce383360e03eb1" gracePeriod=25
Oct 29 23:05:03.337616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.336505    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7" podUID="3b7c5692-52b1-4d25-88fc-4d731525aaeb" containerName="payload-pre-processing" containerID="cri-o://1338acec613e3a575595d88c776eafa82bed1ff05db5fe41f2363c04bf3dd0a1" gracePeriod=25
Oct 29 23:05:03.355960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.355932    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:05:03.356093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.355986    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:05:03.524851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.521112    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:05:03.524851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:03.521172    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:05:04.108792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.091657    4827 generic.go:356] "Generic (PLEG): container finished" podID="3b7c5692-52b1-4d25-88fc-4d731525aaeb" containerID="1338acec613e3a575595d88c776eafa82bed1ff05db5fe41f2363c04bf3dd0a1" exitCode=0
Oct 29 23:05:04.108792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.091934    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7" event={"ID":"3b7c5692-52b1-4d25-88fc-4d731525aaeb","Type":"ContainerDied","Data":"1338acec613e3a575595d88c776eafa82bed1ff05db5fe41f2363c04bf3dd0a1"}
Oct 29 23:05:04.153793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.146495    4827 generic.go:356] "Generic (PLEG): container finished" podID="333138a6-55c4-459d-89d2-f6044d08a481" containerID="9fa0da9b859d641517e405bfc1cc5db650b44fb2f63c6f26ccce383360e03eb1" exitCode=0
Oct 29 23:05:04.154224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.148858    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm" event={"ID":"333138a6-55c4-459d-89d2-f6044d08a481","Type":"ContainerDied","Data":"9fa0da9b859d641517e405bfc1cc5db650b44fb2f63c6f26ccce383360e03eb1"}
Oct 29 23:05:04.201299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.201265    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 23:05:04.201607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.201577    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 23:05:04.257096 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:05:04.253095    4827 logging.go:55] [core] [Channel #2700 SubChannel #2701]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.179:50052", ServerName: "10.134.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.179:50052: connect: connection refused"
Oct 29 23:05:04.358872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.356152    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:05:04.358872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.356207    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:05:04.550597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.548744    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:05:04.550597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.548840    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:05:04.660343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.660269    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:05:04.762192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.762159    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333138a6-55c4-459d-89d2-f6044d08a481-kube-api-access-ltrft\" (UniqueName: \"kubernetes.io/projected/333138a6-55c4-459d-89d2-f6044d08a481-kube-api-access-ltrft\") pod \"333138a6-55c4-459d-89d2-f6044d08a481\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") "
Oct 29 23:05:04.762476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.762459    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333138a6-55c4-459d-89d2-f6044d08a481-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333138a6-55c4-459d-89d2-f6044d08a481-plugins-config-volume\") pod \"333138a6-55c4-459d-89d2-f6044d08a481\" (UID: \"333138a6-55c4-459d-89d2-f6044d08a481\") "
Oct 29 23:05:04.770117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.770083    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333138a6-55c4-459d-89d2-f6044d08a481-plugins-config-volume" pod "333138a6-55c4-459d-89d2-f6044d08a481" (UID: "333138a6-55c4-459d-89d2-f6044d08a481"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:04.920933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.872176    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333138a6-55c4-459d-89d2-f6044d08a481-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:04.942868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:04.876019    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333138a6-55c4-459d-89d2-f6044d08a481-kube-api-access-ltrft" pod "333138a6-55c4-459d-89d2-f6044d08a481" (UID: "333138a6-55c4-459d-89d2-f6044d08a481"). InnerVolumeSpecName "kube-api-access-ltrft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:05.058171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.057647    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltrft\" (UniqueName: \"kubernetes.io/projected/333138a6-55c4-459d-89d2-f6044d08a481-kube-api-access-ltrft\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:05.061528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.061504    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:05:05.211440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.211342    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 23:05:05.211711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.211678    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 23:05:05.252306 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.251930    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7" podUID="3b7c5692-52b1-4d25-88fc-4d731525aaeb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.179:50052\" within 1s: context deadline exceeded"
Oct 29 23:05:05.254568 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.254539    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7" event={"ID":"3b7c5692-52b1-4d25-88fc-4d731525aaeb","Type":"ContainerDied","Data":"17ce362df0867185af11ab6c0361fadc2bcc77f670b0fd08fddb2eb55aec0727"}
Oct 29 23:05:05.255976 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.255900    4827 scope.go:122] "RemoveContainer" containerID="1338acec613e3a575595d88c776eafa82bed1ff05db5fe41f2363c04bf3dd0a1"
Oct 29 23:05:05.256141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.254829    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"
Oct 29 23:05:05.257313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.257297    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm" event={"ID":"333138a6-55c4-459d-89d2-f6044d08a481","Type":"ContainerDied","Data":"b62aaf4fb7129b930a5c1aff78d55cadd82c09785383c4135967a03752835ea6"}
Oct 29 23:05:05.257688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.257479    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"
Oct 29 23:05:05.261307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.261276    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7c5692-52b1-4d25-88fc-4d731525aaeb-kube-api-access-j6n24\" (UniqueName: \"kubernetes.io/projected/3b7c5692-52b1-4d25-88fc-4d731525aaeb-kube-api-access-j6n24\") pod \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") "
Oct 29 23:05:05.261415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.261336    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7c5692-52b1-4d25-88fc-4d731525aaeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7c5692-52b1-4d25-88fc-4d731525aaeb-plugins-config-volume\") pod \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\" (UID: \"3b7c5692-52b1-4d25-88fc-4d731525aaeb\") "
Oct 29 23:05:05.404816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.306857    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7c5692-52b1-4d25-88fc-4d731525aaeb-plugins-config-volume" pod "3b7c5692-52b1-4d25-88fc-4d731525aaeb" (UID: "3b7c5692-52b1-4d25-88fc-4d731525aaeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:05.404816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.311841    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7c5692-52b1-4d25-88fc-4d731525aaeb-kube-api-access-j6n24" pod "3b7c5692-52b1-4d25-88fc-4d731525aaeb" (UID: "3b7c5692-52b1-4d25-88fc-4d731525aaeb"). InnerVolumeSpecName "kube-api-access-j6n24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:05.404816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.365160    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:05:05.404816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.365217    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:05:05.404816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.391835    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6n24\" (UniqueName: \"kubernetes.io/projected/3b7c5692-52b1-4d25-88fc-4d731525aaeb-kube-api-access-j6n24\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:05.404816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.391864    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7c5692-52b1-4d25-88fc-4d731525aaeb-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:05.491804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.484959    4827 scope.go:122] "RemoveContainer" containerID="9fa0da9b859d641517e405bfc1cc5db650b44fb2f63c6f26ccce383360e03eb1"
Oct 29 23:05:05.718865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.546754    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:05:05.718865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:05.689059    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:05:06.014592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.014554    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"]
Oct 29 23:05:06.245793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.244398    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 23:05:06.245793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.244460    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 23:05:06.306458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.306422    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-6b857b65f6-6skjm"]
Oct 29 23:05:06.362382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.362349    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:05:06.362680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.362610    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"]
Oct 29 23:05:06.363043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.362611    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:05:06.363141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.363108    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:05:06.364128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.364084    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:05:06.417792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.392587    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-786f6f8c5f-2vqb7"]
Oct 29 23:05:06.443182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.443128    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:05:06.531133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.531016    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:05:06.531528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.531484    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:05:06.539017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.538983    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333138a6-55c4-459d-89d2-f6044d08a481" path="/var/lib/kubelet/pods/333138a6-55c4-459d-89d2-f6044d08a481/volumes"
Oct 29 23:05:06.539908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:06.539883    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7c5692-52b1-4d25-88fc-4d731525aaeb" path="/var/lib/kubelet/pods/3b7c5692-52b1-4d25-88fc-4d731525aaeb/volumes"
Oct 29 23:05:07.229637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:07.229609    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:05:07.230840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:07.230813    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:05:07.232653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:07.232544    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:05:07.350405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:07.350370    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:05:07.350913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:07.350855    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:05:07.535125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:07.535040    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 23:05:07.535386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:07.535357    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 23:05:08.349535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:08.349495    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 29 23:05:08.349717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:08.349549    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 29 23:05:08.509425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:08.509395    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:05:08.509899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:08.509878    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:05:08.510554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:08.510533    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:05:09.382160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:09.380084    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:05:09.382160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:09.381684    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:05:09.389499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:09.389006    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:05:13.028686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:13.028655    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:05:15.700592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:15.700554    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"]
Oct 29 23:05:15.701092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:15.700783    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd" podUID="35e73970-3f74-470e-9620-fd257b432d9f" containerName="maas-api" containerID="cri-o://abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6" gracePeriod=30
Oct 29 23:05:16.140785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.137871    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:05:16.194419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.193028    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls\") pod \"35e73970-3f74-470e-9620-fd257b432d9f\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") "
Oct 29 23:05:16.194419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.193135    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e73970-3f74-470e-9620-fd257b432d9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e73970-3f74-470e-9620-fd257b432d9f-openshift-service-ca\") pod \"35e73970-3f74-470e-9620-fd257b432d9f\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") "
Oct 29 23:05:16.194419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.193220    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-metrics-certs\") pod \"35e73970-3f74-470e-9620-fd257b432d9f\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") "
Oct 29 23:05:16.194419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.193354    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e73970-3f74-470e-9620-fd257b432d9f-kube-api-access-x9ksv\" (UniqueName: \"kubernetes.io/projected/35e73970-3f74-470e-9620-fd257b432d9f-kube-api-access-x9ksv\") pod \"35e73970-3f74-470e-9620-fd257b432d9f\" (UID: \"35e73970-3f74-470e-9620-fd257b432d9f\") "
Oct 29 23:05:16.197830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.197667    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e73970-3f74-470e-9620-fd257b432d9f-openshift-service-ca" pod "35e73970-3f74-470e-9620-fd257b432d9f" (UID: "35e73970-3f74-470e-9620-fd257b432d9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:16.339843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.323438    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35e73970-3f74-470e-9620-fd257b432d9f-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:16.353817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.350677    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e73970-3f74-470e-9620-fd257b432d9f-kube-api-access-x9ksv" pod "35e73970-3f74-470e-9620-fd257b432d9f" (UID: "35e73970-3f74-470e-9620-fd257b432d9f"). InnerVolumeSpecName "kube-api-access-x9ksv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:16.353817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.351454    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls" pod "35e73970-3f74-470e-9620-fd257b432d9f" (UID: "35e73970-3f74-470e-9620-fd257b432d9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:16.434124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.434038    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-metrics-certs" pod "35e73970-3f74-470e-9620-fd257b432d9f" (UID: "35e73970-3f74-470e-9620-fd257b432d9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:16.435817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.435108    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ksv\" (UniqueName: \"kubernetes.io/projected/35e73970-3f74-470e-9620-fd257b432d9f-kube-api-access-x9ksv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:16.435817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.435145    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:16.435817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.435160    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35e73970-3f74-470e-9620-fd257b432d9f-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:16.622445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.622416    4827 generic.go:356] "Generic (PLEG): container finished" podID="35e73970-3f74-470e-9620-fd257b432d9f" containerID="abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6" exitCode=0
Oct 29 23:05:16.622636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.622464    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd" event={"ID":"35e73970-3f74-470e-9620-fd257b432d9f","Type":"ContainerDied","Data":"abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6"}
Oct 29 23:05:16.622636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.622500    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd" event={"ID":"35e73970-3f74-470e-9620-fd257b432d9f","Type":"ContainerDied","Data":"592b7eb628b809b68b0266b18b951a371900b76847324c789961c88ac6e74d5e"}
Oct 29 23:05:16.622636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.622522    4827 scope.go:122] "RemoveContainer" containerID="abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6"
Oct 29 23:05:16.622859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.622651    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"
Oct 29 23:05:16.648534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.648504    4827 scope.go:122] "RemoveContainer" containerID="abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6"
Oct 29 23:05:16.651428 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:05:16.651396    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6\": container with ID starting with abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6 not found: ID does not exist" containerID="abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6"
Oct 29 23:05:16.651650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.651618    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6"} err="failed to get container status \"abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6\": rpc error: code = NotFound desc = could not find container \"abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6\": container with ID starting with abdc7be79b719782340930988b4b909ce8e06cbb208af2ff1329d4202a3d56f6 not found: ID does not exist"
Oct 29 23:05:16.701805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.699944    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"]
Oct 29 23:05:16.704008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:16.703984    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-53a768-6458bcd556-r8pxd"]
Oct 29 23:05:18.530788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:18.528126    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e73970-3f74-470e-9620-fd257b432d9f" path="/var/lib/kubelet/pods/35e73970-3f74-470e-9620-fd257b432d9f/volumes"
Oct 29 23:05:19.872647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.872615    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"]
Oct 29 23:05:19.873475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873453    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3b7c5692-52b1-4d25-88fc-4d731525aaeb" containerName="payload-pre-processing"
Oct 29 23:05:19.873475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873473    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7c5692-52b1-4d25-88fc-4d731525aaeb" containerName="payload-pre-processing"
Oct 29 23:05:19.873697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873487    4827 cpu_manager.go:414] "containerMap: removing container" podUID="333138a6-55c4-459d-89d2-f6044d08a481" containerName="payload-processing"
Oct 29 23:05:19.873697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873496    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333138a6-55c4-459d-89d2-f6044d08a481" containerName="payload-processing"
Oct 29 23:05:19.873697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873525    4827 cpu_manager.go:414] "containerMap: removing container" podUID="35e73970-3f74-470e-9620-fd257b432d9f" containerName="maas-api"
Oct 29 23:05:19.873697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873533    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e73970-3f74-470e-9620-fd257b432d9f" containerName="maas-api"
Oct 29 23:05:19.873883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873708    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e73970-3f74-470e-9620-fd257b432d9f" containerName="maas-api"
Oct 29 23:05:19.873883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873730    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7c5692-52b1-4d25-88fc-4d731525aaeb" containerName="payload-pre-processing"
Oct 29 23:05:19.873883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.873739    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="333138a6-55c4-459d-89d2-f6044d08a481" containerName="payload-processing"
Oct 29 23:05:19.876898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.876882    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:19.895452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.895430    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4f2e3a-serving-cert\""
Oct 29 23:05:19.909177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.909153    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"]
Oct 29 23:05:19.992050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.992012    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab1abeb-5508-4146-944e-6d48e5ccbc87-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:19.992225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.992175    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2qd\" (UniqueName: \"kubernetes.io/projected/5ab1abeb-5508-4146-944e-6d48e5ccbc87-kube-api-access-cj2qd\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:19.992279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.992227    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:19.992373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:19.992355    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-metrics-certs\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.098951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.098907    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-metrics-certs\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.099286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.099083    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab1abeb-5508-4146-944e-6d48e5ccbc87-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.099286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.099227    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2qd\" (UniqueName: \"kubernetes.io/projected/5ab1abeb-5508-4146-944e-6d48e5ccbc87-kube-api-access-cj2qd\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.099422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.099289    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.102498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.102063    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab1abeb-5508-4146-944e-6d48e5ccbc87-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.125311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.122959    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.126919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.126886    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2qd\" (UniqueName: \"kubernetes.io/projected/5ab1abeb-5508-4146-944e-6d48e5ccbc87-kube-api-access-cj2qd\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.149793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.140597    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-metrics-certs\") pod \"maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.192785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.192735    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:20.563562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.563532    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"]
Oct 29 23:05:20.564333 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:05:20.564308    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab1abeb_5508_4146_944e_6d48e5ccbc87.slice/crio-90f92c75ab5c0fa34bfbeedbec662b14d9cfbccb57a6e61f80945b8e7f5f985f WatchSource:0}: Error finding container 90f92c75ab5c0fa34bfbeedbec662b14d9cfbccb57a6e61f80945b8e7f5f985f: Status 404 returned error can't find the container with id 90f92c75ab5c0fa34bfbeedbec662b14d9cfbccb57a6e61f80945b8e7f5f985f
Oct 29 23:05:20.653729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:20.653697    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh" event={"ID":"5ab1abeb-5508-4146-944e-6d48e5ccbc87","Type":"ContainerStarted","Data":"90f92c75ab5c0fa34bfbeedbec662b14d9cfbccb57a6e61f80945b8e7f5f985f"}
Oct 29 23:05:21.663326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:21.663281    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh" event={"ID":"5ab1abeb-5508-4146-944e-6d48e5ccbc87","Type":"ContainerStarted","Data":"212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5"}
Oct 29 23:05:21.663828 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:21.663358    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:21.705431 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:21.705385    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh" podStartSLOduration=2.329672583 podStartE2EDuration="2.705372375s" totalImagesPullingTime="375.699792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:21.704948078 +0000 UTC m=+1357.327859099" watchObservedRunningTime="2026-10-29 23:05:21.705372375 +0000 UTC m=+1357.328283394"
Oct 29 23:05:27.752790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.741613    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"]
Oct 29 23:05:27.752790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.746462    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:27.770827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.755692    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"]
Oct 29 23:05:27.775601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.775202    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b7950-dockercfg-q4s6g\""
Oct 29 23:05:27.775997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.775973    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b7950\""
Oct 29 23:05:27.807379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.807340    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:05:27.880305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.880260    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrx5\" (UniqueName: \"kubernetes.io/projected/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-kube-api-access-5zrx5\") pod \"payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:27.880710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.880686    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:27.984796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.983490    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:27.984796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.983863    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrx5\" (UniqueName: \"kubernetes.io/projected/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-kube-api-access-5zrx5\") pod \"payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:27.986962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:27.985114    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:28.008011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:28.007939    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrx5\" (UniqueName: \"kubernetes.io/projected/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-kube-api-access-5zrx5\") pod \"payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:28.111046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:28.111002    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:28.473712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:28.473689    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"]
Oct 29 23:05:28.474729 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:05:28.474704    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11eccab3_9bc6_42f9_92c1_0a3a2f349bb0.slice/crio-8dfa10c895b92cbdc49006149464acd9eb6fe897bf500a7ff49b8f65c0976ea7 WatchSource:0}: Error finding container 8dfa10c895b92cbdc49006149464acd9eb6fe897bf500a7ff49b8f65c0976ea7: Status 404 returned error can't find the container with id 8dfa10c895b92cbdc49006149464acd9eb6fe897bf500a7ff49b8f65c0976ea7
Oct 29 23:05:28.798660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:28.798563    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc" event={"ID":"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0","Type":"ContainerStarted","Data":"af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6"}
Oct 29 23:05:28.798660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:28.798631    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:28.798660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:28.798654    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc" event={"ID":"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0","Type":"ContainerStarted","Data":"8dfa10c895b92cbdc49006149464acd9eb6fe897bf500a7ff49b8f65c0976ea7"}
Oct 29 23:05:28.841243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:28.841190    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc" podStartSLOduration=1.841175387 podStartE2EDuration="1.841175387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:28.839456702 +0000 UTC m=+1364.462367726" watchObservedRunningTime="2026-10-29 23:05:28.841175387 +0000 UTC m=+1364.464086499"
Oct 29 23:05:32.825273 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:05:32.825235    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:05:34.670750 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:34.670713    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"]
Oct 29 23:05:34.702378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:34.702343    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"]
Oct 29 23:05:37.456520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:37.456492    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-tjghh_8160871b-24d7-4b6b-bbd5-64ed33ae6382/payload-processing/0.log"
Oct 29 23:05:37.662323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:37.662300    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-tjghh_8160871b-24d7-4b6b-bbd5-64ed33ae6382/payload-processing/0.log"
Oct 29 23:05:37.855294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:37.855222    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt_9afd352a-1d5d-473d-9e13-930a469ad580/payload-processing/0.log"
Oct 29 23:05:38.057806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:38.057746    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt_9afd352a-1d5d-473d-9e13-930a469ad580/payload-processing/0.log"
Oct 29 23:05:39.671861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:39.671803    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc" podUID="fd2f5612-2e40-420a-9648-9d62b745a0f8" containerName="payload-processing" containerID="cri-o://5f20810166f5bbd25dcf4472039af0caaff36ca42987084913a48e79569e8987" gracePeriod=25
Oct 29 23:05:39.703329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:39.703275    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl" podUID="186d20be-496b-4a16-ac9b-f5869c365c0e" containerName="payload-pre-processing" containerID="cri-o://7b808967477fcc0e505f66bacee7b687d76518c0ed5c2f540e0f7445d7da540f" gracePeriod=25
Oct 29 23:05:39.810260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:39.810236    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:05:39.943814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:39.943684    4827 generic.go:356] "Generic (PLEG): container finished" podID="186d20be-496b-4a16-ac9b-f5869c365c0e" containerID="7b808967477fcc0e505f66bacee7b687d76518c0ed5c2f540e0f7445d7da540f" exitCode=0
Oct 29 23:05:39.944104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:39.944076    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl" event={"ID":"186d20be-496b-4a16-ac9b-f5869c365c0e","Type":"ContainerDied","Data":"7b808967477fcc0e505f66bacee7b687d76518c0ed5c2f540e0f7445d7da540f"}
Oct 29 23:05:39.946493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:39.946469    4827 generic.go:356] "Generic (PLEG): container finished" podID="fd2f5612-2e40-420a-9648-9d62b745a0f8" containerID="5f20810166f5bbd25dcf4472039af0caaff36ca42987084913a48e79569e8987" exitCode=0
Oct 29 23:05:39.946664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:39.946640    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc" event={"ID":"fd2f5612-2e40-420a-9648-9d62b745a0f8","Type":"ContainerDied","Data":"5f20810166f5bbd25dcf4472039af0caaff36ca42987084913a48e79569e8987"}
Oct 29 23:05:40.071255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.071224    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:05:40.116443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.116420    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:05:40.193381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.193346    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186d20be-496b-4a16-ac9b-f5869c365c0e-kube-api-access-pg2xz\" (UniqueName: \"kubernetes.io/projected/186d20be-496b-4a16-ac9b-f5869c365c0e-kube-api-access-pg2xz\") pod \"186d20be-496b-4a16-ac9b-f5869c365c0e\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") "
Oct 29 23:05:40.193639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.193502    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2f5612-2e40-420a-9648-9d62b745a0f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2f5612-2e40-420a-9648-9d62b745a0f8-plugins-config-volume\") pod \"fd2f5612-2e40-420a-9648-9d62b745a0f8\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") "
Oct 29 23:05:40.193751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.193722    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186d20be-496b-4a16-ac9b-f5869c365c0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186d20be-496b-4a16-ac9b-f5869c365c0e-plugins-config-volume\") pod \"186d20be-496b-4a16-ac9b-f5869c365c0e\" (UID: \"186d20be-496b-4a16-ac9b-f5869c365c0e\") "
Oct 29 23:05:40.194111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.193932    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2f5612-2e40-420a-9648-9d62b745a0f8-plugins-config-volume" pod "fd2f5612-2e40-420a-9648-9d62b745a0f8" (UID: "fd2f5612-2e40-420a-9648-9d62b745a0f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:40.194201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.194166    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186d20be-496b-4a16-ac9b-f5869c365c0e-plugins-config-volume" pod "186d20be-496b-4a16-ac9b-f5869c365c0e" (UID: "186d20be-496b-4a16-ac9b-f5869c365c0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:40.194301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.194273    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2f5612-2e40-420a-9648-9d62b745a0f8-kube-api-access-b8q5l\" (UniqueName: \"kubernetes.io/projected/fd2f5612-2e40-420a-9648-9d62b745a0f8-kube-api-access-b8q5l\") pod \"fd2f5612-2e40-420a-9648-9d62b745a0f8\" (UID: \"fd2f5612-2e40-420a-9648-9d62b745a0f8\") "
Oct 29 23:05:40.195429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.195405    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2f5612-2e40-420a-9648-9d62b745a0f8-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:40.195537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.195435    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186d20be-496b-4a16-ac9b-f5869c365c0e-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:40.195733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.195706    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186d20be-496b-4a16-ac9b-f5869c365c0e-kube-api-access-pg2xz" pod "186d20be-496b-4a16-ac9b-f5869c365c0e" (UID: "186d20be-496b-4a16-ac9b-f5869c365c0e"). InnerVolumeSpecName "kube-api-access-pg2xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:40.196802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.196783    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2f5612-2e40-420a-9648-9d62b745a0f8-kube-api-access-b8q5l" pod "fd2f5612-2e40-420a-9648-9d62b745a0f8" (UID: "fd2f5612-2e40-420a-9648-9d62b745a0f8"). InnerVolumeSpecName "kube-api-access-b8q5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:40.297393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.297363    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2xz\" (UniqueName: \"kubernetes.io/projected/186d20be-496b-4a16-ac9b-f5869c365c0e-kube-api-access-pg2xz\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:40.297393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.297391    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8q5l\" (UniqueName: \"kubernetes.io/projected/fd2f5612-2e40-420a-9648-9d62b745a0f8-kube-api-access-b8q5l\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:40.955903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.955874    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"
Oct 29 23:05:40.955903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.955895    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl" event={"ID":"186d20be-496b-4a16-ac9b-f5869c365c0e","Type":"ContainerDied","Data":"ff785a604164a68eb2a4dedffa670583010ccaf7278ee5f762685f4401b4edb7"}
Oct 29 23:05:40.956412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.955952    4827 scope.go:122] "RemoveContainer" containerID="7b808967477fcc0e505f66bacee7b687d76518c0ed5c2f540e0f7445d7da540f"
Oct 29 23:05:40.957519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.957499    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"
Oct 29 23:05:40.957616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.957526    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc" event={"ID":"fd2f5612-2e40-420a-9648-9d62b745a0f8","Type":"ContainerDied","Data":"e38f3d38018c585e332bdb638f4947578807e1704642b86e6fbc69979796dee7"}
Oct 29 23:05:40.967342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:40.967323    4827 scope.go:122] "RemoveContainer" containerID="5f20810166f5bbd25dcf4472039af0caaff36ca42987084913a48e79569e8987"
Oct 29 23:05:41.008847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.008820    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"]
Oct 29 23:05:41.020697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.020672    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c73f1d-578fd8f65d-qgrlc"]
Oct 29 23:05:41.046638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.046602    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"]
Oct 29 23:05:41.050385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.050362    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c73f1d-7f46fc56c9-f4rzl"]
Oct 29 23:05:41.267333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.267259    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"]
Oct 29 23:05:41.267986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.267961    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fd2f5612-2e40-420a-9648-9d62b745a0f8" containerName="payload-processing"
Oct 29 23:05:41.268056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.267986    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2f5612-2e40-420a-9648-9d62b745a0f8" containerName="payload-processing"
Oct 29 23:05:41.268056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.268020    4827 cpu_manager.go:414] "containerMap: removing container" podUID="186d20be-496b-4a16-ac9b-f5869c365c0e" containerName="payload-pre-processing"
Oct 29 23:05:41.268056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.268025    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186d20be-496b-4a16-ac9b-f5869c365c0e" containerName="payload-pre-processing"
Oct 29 23:05:41.268162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.268144    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="186d20be-496b-4a16-ac9b-f5869c365c0e" containerName="payload-pre-processing"
Oct 29 23:05:41.268162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.268159    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2f5612-2e40-420a-9648-9d62b745a0f8" containerName="payload-processing"
Oct 29 23:05:41.271268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.271246    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.292850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.286545    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-77238f-openshift-default-dockercfg-24287\""
Oct 29 23:05:41.295790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.295748    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"]
Oct 29 23:05:41.434788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.434294    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.434788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.434540    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.434788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.434623    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46clt\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-kube-api-access-46clt\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.434788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.434667    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.434788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.434755    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.435228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.434868    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.435228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.434942    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.435228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.435030    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.435228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.435119    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.544852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.539750    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.545178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.545149    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clt\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-kube-api-access-46clt\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.545332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.545317    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.545509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.545492    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.545691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.545676    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.545870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.545852    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.546076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.546059    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.556789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.546242    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.557089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.557064    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.557431 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.549877    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.557563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.550122    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.557659 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.550677    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.557753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.549548    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.558163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.558142    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.632484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.632407    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.634225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.634178    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.635521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.635475    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46clt\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-kube-api-access-46clt\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.662795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.657387    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:41.888498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:41.888467    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:42.212895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:42.212862    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"]
Oct 29 23:05:42.213821 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:05:42.213761    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739f2a27_1668_47cf_9b3e_5beab9d7cf1d.slice/crio-466f58b59563706d042dcac00f4dde7427e4ad50edcb811668ad1e2c53127a8a WatchSource:0}: Error finding container 466f58b59563706d042dcac00f4dde7427e4ad50edcb811668ad1e2c53127a8a: Status 404 returned error can't find the container with id 466f58b59563706d042dcac00f4dde7427e4ad50edcb811668ad1e2c53127a8a
Oct 29 23:05:42.216801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:42.216749    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:42.217070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:42.216834    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:42.217070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:42.216866    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:42.492740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:42.492661    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186d20be-496b-4a16-ac9b-f5869c365c0e" path="/var/lib/kubelet/pods/186d20be-496b-4a16-ac9b-f5869c365c0e/volumes"
Oct 29 23:05:42.493304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:42.493284    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2f5612-2e40-420a-9648-9d62b745a0f8" path="/var/lib/kubelet/pods/fd2f5612-2e40-420a-9648-9d62b745a0f8/volumes"
Oct 29 23:05:43.009830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.008359    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"]
Oct 29 23:05:43.013636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.013610    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.016334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.015599    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" event={"ID":"739f2a27-1668-47cf-9b3e-5beab9d7cf1d","Type":"ContainerStarted","Data":"e2df71419852a05589c3650983941fc519f123096b9afb8afe52118cb159a18a"}
Oct 29 23:05:43.016469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.016362    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" event={"ID":"739f2a27-1668-47cf-9b3e-5beab9d7cf1d","Type":"ContainerStarted","Data":"466f58b59563706d042dcac00f4dde7427e4ad50edcb811668ad1e2c53127a8a"}
Oct 29 23:05:43.016469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.016395    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"]
Oct 29 23:05:43.065391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.065328    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podStartSLOduration=2.065310457 podStartE2EDuration="2.065310457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:43.059912251 +0000 UTC m=+1378.682823272" watchObservedRunningTime="2026-10-29 23:05:43.065310457 +0000 UTC m=+1378.688221481"
Oct 29 23:05:43.093022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.092995    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093143    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e343d-7c51-411b-8127-547323e065c0-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093204    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093402    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e343d-7c51-411b-8127-547323e065c0-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093459    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093496    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093630    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093657    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngps\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-kube-api-access-vngps\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.093713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.093693    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.205530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205484    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.205705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205598    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e343d-7c51-411b-8127-547323e065c0-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.205705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205647    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.205924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205734    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e343d-7c51-411b-8127-547323e065c0-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.205924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205804    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.205924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205843    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.206087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205923    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.206087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205947    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngps\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-kube-api-access-vngps\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.206087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.205992    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.206478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.206452    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.206917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.206889    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.209689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.209106    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.209689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.209591    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.210346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.210318    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e343d-7c51-411b-8127-547323e065c0-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.212720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.212696    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.229787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.222529    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e343d-7c51-411b-8127-547323e065c0-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.253644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.252038    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngps\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-kube-api-access-vngps\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.274396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.268658    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.336482 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.336454    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:43.660684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.660655    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"]
Oct 29 23:05:43.660994 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:05:43.660973    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43e343d_7c51_411b_8127_547323e065c0.slice/crio-6456b50f2035c3840e8d9e01076dec05c4e148be1c06cee2309dbf6ba456eea2 WatchSource:0}: Error finding container 6456b50f2035c3840e8d9e01076dec05c4e148be1c06cee2309dbf6ba456eea2: Status 404 returned error can't find the container with id 6456b50f2035c3840e8d9e01076dec05c4e148be1c06cee2309dbf6ba456eea2
Oct 29 23:05:43.664062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.664030    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:43.664119 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.664095    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:43.664153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.664125    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:43.889177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.889141    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:43.895475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:43.895450    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:44.026675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.026574    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" event={"ID":"e43e343d-7c51-411b-8127-547323e065c0","Type":"ContainerStarted","Data":"7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5"}
Oct 29 23:05:44.026675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.026635    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" event={"ID":"e43e343d-7c51-411b-8127-547323e065c0","Type":"ContainerStarted","Data":"6456b50f2035c3840e8d9e01076dec05c4e148be1c06cee2309dbf6ba456eea2"}
Oct 29 23:05:44.027029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.026999    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:44.027894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.027876    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:05:44.077184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.077122    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podStartSLOduration=2.077103038 podStartE2EDuration="2.077103038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:44.070531334 +0000 UTC m=+1379.693442373" watchObservedRunningTime="2026-10-29 23:05:44.077103038 +0000 UTC m=+1379.700014059"
Oct 29 23:05:44.337167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.337089    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:44.339504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.339479    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 23:05:44.339630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:44.339523    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 23:05:45.337692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:45.337664    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 23:05:45.338091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:45.337717    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 23:05:46.337337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.337301    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 23:05:46.337506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.337369    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 23:05:46.894047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.893988    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"]
Oct 29 23:05:46.898290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.898270    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:46.899035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.899012    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"]
Oct 29 23:05:46.902044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.902025    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:46.919841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.919818    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"]
Oct 29 23:05:46.922161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:46.922140    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"]
Oct 29 23:05:47.000507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000465    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.000507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000514    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.000791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000537    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.000791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000649    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.000791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000691    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.000983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000841    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqbvn\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-kube-api-access-bqbvn\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.000983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000913    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.000983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.000966    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.001142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001045    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.001202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001142    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.001202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001171    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.001202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001190    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5stn\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-kube-api-access-m5stn\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.001360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001205    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.001420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001364    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.001475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001431    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.001530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001481    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.001603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001580    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3bda7f-64d7-41bb-9649-bba9983a995a-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.001657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.001634    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.103748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.103701    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.103980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.103801    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.103980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.103843    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.103980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.103928    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.103980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.103962    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.103993    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5stn\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-kube-api-access-m5stn\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104017    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.104219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104092    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104148    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104159    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104197    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104255    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3bda7f-64d7-41bb-9649-bba9983a995a-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104292    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104360    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104404    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104421    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104439    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104464    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.104540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104520    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.105015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104563    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqbvn\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-kube-api-access-bqbvn\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.105015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.104713    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.105456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.105426    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.105900    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.106121    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.107015    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.107309    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.107530    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.108136    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3bda7f-64d7-41bb-9649-bba9983a995a-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.110874    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.113809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.111051    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.116139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.115671    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.117385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.117361    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.139964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.139933    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqbvn\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-kube-api-access-bqbvn\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.140838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.140811    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5stn\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-kube-api-access-m5stn\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.141226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.141205    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.142112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.142089    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.223607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.223516    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:47.246629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.246601    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:47.339132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.338641    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 23:05:47.339132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.338711    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 23:05:47.882888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.863965    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"]
Oct 29 23:05:47.909041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.908926    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:47.909041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.909027    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:47.909560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:47.909075    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:48.008245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.008217    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"]
Oct 29 23:05:48.009517 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:05:48.009487    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772c47d7_cc86_4369_9921_7fbe19d2a3bd.slice/crio-92cd7f6f0fddaf9659e173ee8a391f7b9f452690977d955f6293335b5a314594 WatchSource:0}: Error finding container 92cd7f6f0fddaf9659e173ee8a391f7b9f452690977d955f6293335b5a314594: Status 404 returned error can't find the container with id 92cd7f6f0fddaf9659e173ee8a391f7b9f452690977d955f6293335b5a314594
Oct 29 23:05:48.013109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.013067    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:48.013216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.013163    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:48.013258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.013217    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:05:48.082303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.082261    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" event={"ID":"7e3bda7f-64d7-41bb-9649-bba9983a995a","Type":"ContainerStarted","Data":"fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8"}
Oct 29 23:05:48.082582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.082563    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" event={"ID":"7e3bda7f-64d7-41bb-9649-bba9983a995a","Type":"ContainerStarted","Data":"51cd5fe616b71f364da75a916eaf910960a54c015a9ec909f130e555f92b7843"}
Oct 29 23:05:48.100652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.100619    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" event={"ID":"772c47d7-cc86-4369-9921-7fbe19d2a3bd","Type":"ContainerStarted","Data":"92cd7f6f0fddaf9659e173ee8a391f7b9f452690977d955f6293335b5a314594"}
Oct 29 23:05:48.134347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.134269    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podStartSLOduration=2.134248971 podStartE2EDuration="2.134248971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:48.130586856 +0000 UTC m=+1383.753497914" watchObservedRunningTime="2026-10-29 23:05:48.134248971 +0000 UTC m=+1383.757159996"
Oct 29 23:05:48.224448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.224403    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:48.338181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.338140    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 23:05:48.338357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.338218    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 23:05:48.466852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.463333    4827 scope.go:122] "RemoveContainer" containerID="cea57decfdfeaa6bc924231e87d686d685d770e7760152f93bef44a59d2b43e1"
Oct 29 23:05:48.499988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.499952    4827 scope.go:122] "RemoveContainer" containerID="48dbab585e7ca4d2bb7c9c3b1f9b75f6e15fee4955b580c04f9ff31edc043d90"
Oct 29 23:05:48.512938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:48.512908    4827 scope.go:122] "RemoveContainer" containerID="efc73469de74bef1ea4bbe9bd6b0559b14707f51359e2fe2fda037c9eae8f430"
Oct 29 23:05:49.272845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:49.261138    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:05:49.272845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:49.261196    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:05:49.342790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:49.342160    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" event={"ID":"772c47d7-cc86-4369-9921-7fbe19d2a3bd","Type":"ContainerStarted","Data":"d1463cac3feb50d4c67765aac7ccef3d91a194b07c2f23bf424549fd23bbd714"}
Oct 29 23:05:49.342790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:49.342600    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 23:05:49.342790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:49.342646    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 23:05:50.224889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.224851    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:05:50.225107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.224916    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:05:50.247369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.247334    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:50.248857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.248820    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 29 23:05:50.248990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.248874    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 29 23:05:50.342459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.342430    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:50.343092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.342968    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:50.343627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.343608    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:05:50.418111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.418061    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podStartSLOduration=4.418046255 podStartE2EDuration="4.418046255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:49.425190107 +0000 UTC m=+1385.048101128" watchObservedRunningTime="2026-10-29 23:05:50.418046255 +0000 UTC m=+1386.040957274"
Oct 29 23:05:50.828015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.827961    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"]
Oct 29 23:05:50.829851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:50.828226    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" podUID="3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" containerName="maas-api" containerID="cri-o://a664fbb31931a1fe5e19008fdfa8db6290942d78e282b887e8626c84910e0c26" gracePeriod=30
Oct 29 23:05:51.229219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:51.229185    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:05:51.229483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:51.229458    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:05:51.254523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:51.254492    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 29 23:05:51.254848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:51.254814    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 29 23:05:51.471792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:51.471289    4827 generic.go:356] "Generic (PLEG): container finished" podID="3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" containerID="a664fbb31931a1fe5e19008fdfa8db6290942d78e282b887e8626c84910e0c26" exitCode=0
Oct 29 23:05:51.473607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:51.472530    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" event={"ID":"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe","Type":"ContainerDied","Data":"a664fbb31931a1fe5e19008fdfa8db6290942d78e282b887e8626c84910e0c26"}
Oct 29 23:05:52.209497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.209408    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:05:52.236707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.233921    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:05:52.236707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.234002    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:05:52.280707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.280674    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 29 23:05:52.280973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.280946    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 29 23:05:52.325788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.320272    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls\") pod \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") "
Oct 29 23:05:52.325788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.320618    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-openshift-service-ca\") pod \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") "
Oct 29 23:05:52.325788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.320720    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-kube-api-access-dtnvq\" (UniqueName: \"kubernetes.io/projected/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-kube-api-access-dtnvq\") pod \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") "
Oct 29 23:05:52.325788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.320760    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-metrics-certs\") pod \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\" (UID: \"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe\") "
Oct 29 23:05:52.325788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.323151    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-openshift-service-ca" pod "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" (UID: "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:52.358016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.357983    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-kube-api-access-dtnvq" pod "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" (UID: "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe"). InnerVolumeSpecName "kube-api-access-dtnvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:52.359261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.359224    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-metrics-certs" pod "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" (UID: "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:52.359513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.359485    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls" pod "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" (UID: "3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:52.423154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.422918    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:52.423154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.422953    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:52.423154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.422969    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtnvq\" (UniqueName: \"kubernetes.io/projected/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-kube-api-access-dtnvq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:52.423154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.422984    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:05:52.535052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.533994    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"
Oct 29 23:05:52.536008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.535975    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" event={"ID":"3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe","Type":"ContainerDied","Data":"6701e509a9359f47bfda2a1550c5ebaeba7de10b991c48177654eb08f1362ede"}
Oct 29 23:05:52.536143 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.536040    4827 scope.go:122] "RemoveContainer" containerID="a664fbb31931a1fe5e19008fdfa8db6290942d78e282b887e8626c84910e0c26"
Oct 29 23:05:52.623046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.623014    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"]
Oct 29 23:05:52.628827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:52.628795    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64"]
Oct 29 23:05:53.224090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:53.224052    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:05:53.224302 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:53.224106    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:05:53.247107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:53.247074    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 29 23:05:53.247266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:53.247128    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 29 23:05:54.226157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:54.225467    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:05:54.226157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:54.225539    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:05:54.247096 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:54.247066    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 29 23:05:54.247229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:54.247146    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 29 23:05:54.656788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:54.649881    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" path="/var/lib/kubelet/pods/3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe/volumes"
Oct 29 23:05:54.848365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:54.848321    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c73f1d-84fff8b5f9-hkd64" podUID="3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.185:8443/health\": context deadline exceeded"
Oct 29 23:05:55.224354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:55.224317    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:05:55.224627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:55.224408    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:05:55.247161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:55.247127    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 29 23:05:55.247536 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:55.247191    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 29 23:05:56.814329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:56.813527    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:56.815201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:56.815176    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:56.816117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:56.816093    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:05:56.866077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:56.866044    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:56.866417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:56.866392    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:56.867364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:56.867346    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:05:58.929620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:58.929585    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"]
Oct 29 23:05:58.931257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:05:58.929864    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy" containerID="cri-o://6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667" gracePeriod=30
Oct 29 23:06:02.827257 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:06:02.827224    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:06:04.541960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.541937    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:06:04.661854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.661816    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-podinfo\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.661878    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-data\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.661945    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-kube-api-access-5vzvl\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-kube-api-access-5vzvl\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662016    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-token\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-token\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662087    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-credential-socket\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662151    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-certs\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662181    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-data" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:06:04.662372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662208    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-envoy\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662285    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5dacfa-5542-4a7d-a809-f02354a36829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5dacfa-5542-4a7d-a809-f02354a36829-istiod-ca-cert\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662316    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-socket\") pod \"ad5dacfa-5542-4a7d-a809-f02354a36829\" (UID: \"ad5dacfa-5542-4a7d-a809-f02354a36829\") "
Oct 29 23:06:04.662597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662416    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-credential-socket" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:06:04.662597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662430    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-certs" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:06:04.662755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.662734    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5dacfa-5542-4a7d-a809-f02354a36829-istiod-ca-cert" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:04.663031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.663007    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-socket" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:06:04.663211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.663147    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5dacfa-5542-4a7d-a809-f02354a36829-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.663334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.663224    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.663334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.663247    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.663334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.663264    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.664394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.664367    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-podinfo" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:06:04.664477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.664403    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-kube-api-access-5vzvl" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "kube-api-access-5vzvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:04.664477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.664425    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-token" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:04.664620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.664600    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-envoy" pod "ad5dacfa-5542-4a7d-a809-f02354a36829" (UID: "ad5dacfa-5542-4a7d-a809-f02354a36829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:06:04.764647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.764604    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.764647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.764641    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vzvl\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-kube-api-access-5vzvl\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.764647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.764654    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.764977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.764666    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.764977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.764678    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5dacfa-5542-4a7d-a809-f02354a36829-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.770543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.770516    4827 generic.go:356] "Generic (PLEG): container finished" podID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerID="6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667" exitCode=0
Oct 29 23:06:04.770674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.770575    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"
Oct 29 23:06:04.770674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.770604    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" event={"ID":"ad5dacfa-5542-4a7d-a809-f02354a36829","Type":"ContainerDied","Data":"6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667"}
Oct 29 23:06:04.770674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.770660    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w" event={"ID":"ad5dacfa-5542-4a7d-a809-f02354a36829","Type":"ContainerDied","Data":"53258a0f5eb76b82956099e339bc1407dc274c616c38565d89dcf7b6f977676f"}
Oct 29 23:06:04.770851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.770685    4827 scope.go:122] "RemoveContainer" containerID="6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667"
Oct 29 23:06:04.781661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.781646    4827 scope.go:122] "RemoveContainer" containerID="75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda"
Oct 29 23:06:04.791521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.791505    4827 scope.go:122] "RemoveContainer" containerID="6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667"
Oct 29 23:06:04.791823 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:06:04.791804    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667\": container with ID starting with 6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667 not found: ID does not exist" containerID="6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667"
Oct 29 23:06:04.791901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.791831    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667"} err="failed to get container status \"6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667\": rpc error: code = NotFound desc = could not find container \"6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667\": container with ID starting with 6b82cb7d5e4fa213f88316ea2557d385832a5e4394335b8f7eddff59b09a1667 not found: ID does not exist"
Oct 29 23:06:04.791901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.791847    4827 scope.go:122] "RemoveContainer" containerID="75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda"
Oct 29 23:06:04.792086 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:06:04.792071    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda\": container with ID starting with 75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda not found: ID does not exist" containerID="75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda"
Oct 29 23:06:04.792134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.792090    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda"} err="failed to get container status \"75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda\": rpc error: code = NotFound desc = could not find container \"75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda\": container with ID starting with 75314954a13210a18196f330cda7c3280cd501b80b4634d130dbbcf9c0f5abda not found: ID does not exist"
Oct 29 23:06:04.822984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.822757    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"]
Oct 29 23:06:04.826417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:04.826398    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c73f1d-openshift-default-fdc6fc589-5sh2w"]
Oct 29 23:06:06.489140 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:06.489105    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" path="/var/lib/kubelet/pods/ad5dacfa-5542-4a7d-a809-f02354a36829/volumes"
Oct 29 23:06:11.351567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:11.351532    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"]
Oct 29 23:06:16.375787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.374842    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5" podUID="96516365-f87e-456e-85dd-dcda19696285" containerName="payload-pre-processing" containerID="cri-o://2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3" gracePeriod=25
Oct 29 23:06:16.780837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.780813    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:06:16.891550 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.891503    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96516365-f87e-456e-85dd-dcda19696285-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96516365-f87e-456e-85dd-dcda19696285-plugins-config-volume\") pod \"96516365-f87e-456e-85dd-dcda19696285\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") "
Oct 29 23:06:16.891845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.891815    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96516365-f87e-456e-85dd-dcda19696285-kube-api-access-r72zb\" (UniqueName: \"kubernetes.io/projected/96516365-f87e-456e-85dd-dcda19696285-kube-api-access-r72zb\") pod \"96516365-f87e-456e-85dd-dcda19696285\" (UID: \"96516365-f87e-456e-85dd-dcda19696285\") "
Oct 29 23:06:16.891987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.891955    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96516365-f87e-456e-85dd-dcda19696285-plugins-config-volume" pod "96516365-f87e-456e-85dd-dcda19696285" (UID: "96516365-f87e-456e-85dd-dcda19696285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:16.892526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.892503    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96516365-f87e-456e-85dd-dcda19696285-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:16.894895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.894864    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96516365-f87e-456e-85dd-dcda19696285-kube-api-access-r72zb" pod "96516365-f87e-456e-85dd-dcda19696285" (UID: "96516365-f87e-456e-85dd-dcda19696285"). InnerVolumeSpecName "kube-api-access-r72zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:16.996787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:16.994521    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72zb\" (UniqueName: \"kubernetes.io/projected/96516365-f87e-456e-85dd-dcda19696285-kube-api-access-r72zb\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:17.195540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.195502    4827 generic.go:356] "Generic (PLEG): container finished" podID="96516365-f87e-456e-85dd-dcda19696285" containerID="2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3" exitCode=0
Oct 29 23:06:17.195729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.195547    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"
Oct 29 23:06:17.195729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.195580    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5" event={"ID":"96516365-f87e-456e-85dd-dcda19696285","Type":"ContainerDied","Data":"2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3"}
Oct 29 23:06:17.195729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.195641    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5" event={"ID":"96516365-f87e-456e-85dd-dcda19696285","Type":"ContainerDied","Data":"b8d63b7891564a0b9e41e9e68b41ae2e64dab72803103a1c11ae87598d0858a0"}
Oct 29 23:06:17.195729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.195684    4827 scope.go:122] "RemoveContainer" containerID="2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3"
Oct 29 23:06:17.209513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.209492    4827 scope.go:122] "RemoveContainer" containerID="2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3"
Oct 29 23:06:17.209889 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:06:17.209869    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3\": container with ID starting with 2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3 not found: ID does not exist" containerID="2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3"
Oct 29 23:06:17.209984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.209900    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3"} err="failed to get container status \"2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3\": rpc error: code = NotFound desc = could not find container \"2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3\": container with ID starting with 2683bf43e0dadbfc0a6b695d449bc541e252c83fd3564d7b3b9be69bf79579b3 not found: ID does not exist"
Oct 29 23:06:17.312804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.310845    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"]
Oct 29 23:06:17.345796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.334282    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-14debc-86bf89b7f9-q6qb5"]
Oct 29 23:06:17.954834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:17.931290    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"]
Oct 29 23:06:18.050168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:18.050123    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f" podUID="f98991aa-3271-4b69-9b1a-191af1e3a47c" containerName="maas-api" containerID="cri-o://1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a" gracePeriod=30
Oct 29 23:06:18.578021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:18.577985    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96516365-f87e-456e-85dd-dcda19696285" path="/var/lib/kubelet/pods/96516365-f87e-456e-85dd-dcda19696285/volumes"
Oct 29 23:06:19.193302 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.192971    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:06:19.296818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.294592    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-metrics-certs\") pod \"f98991aa-3271-4b69-9b1a-191af1e3a47c\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") "
Oct 29 23:06:19.296818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.295117    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98991aa-3271-4b69-9b1a-191af1e3a47c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98991aa-3271-4b69-9b1a-191af1e3a47c-openshift-service-ca\") pod \"f98991aa-3271-4b69-9b1a-191af1e3a47c\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") "
Oct 29 23:06:19.296818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.295223    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98991aa-3271-4b69-9b1a-191af1e3a47c-kube-api-access-sz7nk\" (UniqueName: \"kubernetes.io/projected/f98991aa-3271-4b69-9b1a-191af1e3a47c-kube-api-access-sz7nk\") pod \"f98991aa-3271-4b69-9b1a-191af1e3a47c\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") "
Oct 29 23:06:19.296818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.295369    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls\") pod \"f98991aa-3271-4b69-9b1a-191af1e3a47c\" (UID: \"f98991aa-3271-4b69-9b1a-191af1e3a47c\") "
Oct 29 23:06:19.297622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.297567    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98991aa-3271-4b69-9b1a-191af1e3a47c-openshift-service-ca" pod "f98991aa-3271-4b69-9b1a-191af1e3a47c" (UID: "f98991aa-3271-4b69-9b1a-191af1e3a47c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:19.311016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.310980    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98991aa-3271-4b69-9b1a-191af1e3a47c-kube-api-access-sz7nk" pod "f98991aa-3271-4b69-9b1a-191af1e3a47c" (UID: "f98991aa-3271-4b69-9b1a-191af1e3a47c"). InnerVolumeSpecName "kube-api-access-sz7nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:19.318956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.318910    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls" pod "f98991aa-3271-4b69-9b1a-191af1e3a47c" (UID: "f98991aa-3271-4b69-9b1a-191af1e3a47c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:19.426600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.426553    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-metrics-certs" pod "f98991aa-3271-4b69-9b1a-191af1e3a47c" (UID: "f98991aa-3271-4b69-9b1a-191af1e3a47c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:19.434175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.431868    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:19.434175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.431905    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98991aa-3271-4b69-9b1a-191af1e3a47c-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:19.434175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.431920    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz7nk\" (UniqueName: \"kubernetes.io/projected/f98991aa-3271-4b69-9b1a-191af1e3a47c-kube-api-access-sz7nk\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:19.434175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.431933    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f98991aa-3271-4b69-9b1a-191af1e3a47c-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:19.447817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.447512    4827 generic.go:356] "Generic (PLEG): container finished" podID="f98991aa-3271-4b69-9b1a-191af1e3a47c" containerID="1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a" exitCode=0
Oct 29 23:06:19.447817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.447744    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f" event={"ID":"f98991aa-3271-4b69-9b1a-191af1e3a47c","Type":"ContainerDied","Data":"1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a"}
Oct 29 23:06:19.447817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.447817    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f" event={"ID":"f98991aa-3271-4b69-9b1a-191af1e3a47c","Type":"ContainerDied","Data":"7bf5c4f7e2c63e579909b7774640f7a44d2d92edb30c8381b34fedcf730dc4e9"}
Oct 29 23:06:19.448121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.447853    4827 scope.go:122] "RemoveContainer" containerID="1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a"
Oct 29 23:06:19.448121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.447973    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"
Oct 29 23:06:19.465045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.465014    4827 scope.go:122] "RemoveContainer" containerID="1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a"
Oct 29 23:06:19.465350 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:06:19.465328    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a\": container with ID starting with 1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a not found: ID does not exist" containerID="1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a"
Oct 29 23:06:19.465437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.465358    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a"} err="failed to get container status \"1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a\": rpc error: code = NotFound desc = could not find container \"1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a\": container with ID starting with 1d43288d1f85681f337857895763ebda45e84a911f2e497552e7fce76e6c3a4a not found: ID does not exist"
Oct 29 23:06:19.513317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.513283    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"]
Oct 29 23:06:19.518848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:19.518302    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-14debc-7dd4db4f5-7wg6f"]
Oct 29 23:06:20.490108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:20.490069    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98991aa-3271-4b69-9b1a-191af1e3a47c" path="/var/lib/kubelet/pods/f98991aa-3271-4b69-9b1a-191af1e3a47c/volumes"
Oct 29 23:06:22.475055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.474968    4827 generic.go:356] "Generic (PLEG): container finished" podID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerID="b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d" exitCode=0
Oct 29 23:06:22.475055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.475038    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" event={"ID":"4b4230c6-f355-462c-8a8f-eb917819bd34","Type":"ContainerDied","Data":"b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d"}
Oct 29 23:06:22.475589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.475536    4827 scope.go:122] "RemoveContainer" containerID="b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d"
Oct 29 23:06:22.476881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.476858    4827 generic.go:356] "Generic (PLEG): container finished" podID="f062884e-7973-4797-854e-d2322a868e86" containerID="299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a" exitCode=0
Oct 29 23:06:22.476975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.476902    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" event={"ID":"f062884e-7973-4797-854e-d2322a868e86","Type":"ContainerDied","Data":"299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a"}
Oct 29 23:06:22.477278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.477263    4827 scope.go:122] "RemoveContainer" containerID="299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a"
Oct 29 23:06:22.477469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.477442    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:22.477527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.477514    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:22.477580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.477551    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:22.478959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.478932    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:22.479031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.478999    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:22.479072 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:22.479032    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:23.486475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.486443    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" event={"ID":"f062884e-7973-4797-854e-d2322a868e86","Type":"ContainerStarted","Data":"ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed"}
Oct 29 23:06:23.488519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.488497    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" event={"ID":"4b4230c6-f355-462c-8a8f-eb917819bd34","Type":"ContainerStarted","Data":"09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f"}
Oct 29 23:06:23.799852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.797168    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:06:23.802929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.802890    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 23:06:23.803222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.803194    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 23:06:23.923691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.923653    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:06:23.923939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.923915    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 23:06:23.924025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:23.923968    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 23:06:24.797259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:24.797221    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 23:06:24.797929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:24.797892    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 23:06:24.924815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:24.923570    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 23:06:24.924815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:24.923626    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 23:06:25.799790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:25.799686    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 23:06:25.800328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:25.799749    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 23:06:25.923683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:25.923642    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 23:06:25.923882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:25.923708    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 23:06:26.799731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:26.799692    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 23:06:26.800151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:26.800120    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 23:06:26.930300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:26.930266    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 23:06:26.930611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:26.930576    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 23:06:27.729876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:27.729839    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:06:27.798244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:27.798204    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 23:06:27.798544 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:27.798518    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 23:06:27.924105 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:27.924077    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 23:06:27.924503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:27.924140    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 23:06:28.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:28.800184    4827 patch_prober.go:29] interesting pod/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 23:06:28.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:28.800262    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 23:06:28.923159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:28.923125    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 23:06:28.923356 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:28.923189    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 23:06:29.246369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:29.246330    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:06:29.802149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:29.802121    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:06:29.803021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:29.802999    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:06:29.934815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:29.931210    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:06:29.934815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:29.932570    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:06:32.824672 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:06:32.824634    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:06:40.577979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.577949    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"]
Oct 29 23:06:40.578456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.578167    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97" podUID="b50ad01a-0312-452c-997f-e928048a2c06" containerName="maas-api" containerID="cri-o://4bb28726cdd9c1d5d3d459cd93882b8bacd343db174d84472615108f975d0232" gracePeriod=30
Oct 29 23:06:40.751894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.751862    4827 generic.go:356] "Generic (PLEG): container finished" podID="b50ad01a-0312-452c-997f-e928048a2c06" containerID="4bb28726cdd9c1d5d3d459cd93882b8bacd343db174d84472615108f975d0232" exitCode=0
Oct 29 23:06:40.752083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.751918    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97" event={"ID":"b50ad01a-0312-452c-997f-e928048a2c06","Type":"ContainerDied","Data":"4bb28726cdd9c1d5d3d459cd93882b8bacd343db174d84472615108f975d0232"}
Oct 29 23:06:40.881036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.881012    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:06:40.892580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.892552    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-maas-api-tls\") pod \"b50ad01a-0312-452c-997f-e928048a2c06\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") "
Oct 29 23:06:40.892791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.892757    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-metrics-certs\") pod \"b50ad01a-0312-452c-997f-e928048a2c06\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") "
Oct 29 23:06:40.892901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.892884    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50ad01a-0312-452c-997f-e928048a2c06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50ad01a-0312-452c-997f-e928048a2c06-openshift-service-ca\") pod \"b50ad01a-0312-452c-997f-e928048a2c06\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") "
Oct 29 23:06:40.892975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.892953    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50ad01a-0312-452c-997f-e928048a2c06-kube-api-access-sdvnd\" (UniqueName: \"kubernetes.io/projected/b50ad01a-0312-452c-997f-e928048a2c06-kube-api-access-sdvnd\") pod \"b50ad01a-0312-452c-997f-e928048a2c06\" (UID: \"b50ad01a-0312-452c-997f-e928048a2c06\") "
Oct 29 23:06:40.893222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.893201    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50ad01a-0312-452c-997f-e928048a2c06-openshift-service-ca" pod "b50ad01a-0312-452c-997f-e928048a2c06" (UID: "b50ad01a-0312-452c-997f-e928048a2c06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:40.893538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.893518    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50ad01a-0312-452c-997f-e928048a2c06-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:40.894646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.894619    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-maas-api-tls" pod "b50ad01a-0312-452c-997f-e928048a2c06" (UID: "b50ad01a-0312-452c-997f-e928048a2c06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:40.894729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.894708    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-metrics-certs" pod "b50ad01a-0312-452c-997f-e928048a2c06" (UID: "b50ad01a-0312-452c-997f-e928048a2c06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:40.894944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.894925    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50ad01a-0312-452c-997f-e928048a2c06-kube-api-access-sdvnd" pod "b50ad01a-0312-452c-997f-e928048a2c06" (UID: "b50ad01a-0312-452c-997f-e928048a2c06"). InnerVolumeSpecName "kube-api-access-sdvnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:40.995058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.995027    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:40.995058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.995053    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvnd\" (UniqueName: \"kubernetes.io/projected/b50ad01a-0312-452c-997f-e928048a2c06-kube-api-access-sdvnd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:40.995058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:40.995063    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50ad01a-0312-452c-997f-e928048a2c06-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:41.772883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:41.772577    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97" event={"ID":"b50ad01a-0312-452c-997f-e928048a2c06","Type":"ContainerDied","Data":"908eb58a3168178b2964ea3360d86e989e195daaaebf60d0b045b9614a3950df"}
Oct 29 23:06:41.772883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:41.772645    4827 scope.go:122] "RemoveContainer" containerID="4bb28726cdd9c1d5d3d459cd93882b8bacd343db174d84472615108f975d0232"
Oct 29 23:06:41.772883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:41.772883    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"
Oct 29 23:06:41.824318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:41.824292    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"]
Oct 29 23:06:41.836246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:41.836225    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d18f7-78bc57999c-2fn97"]
Oct 29 23:06:42.489421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:42.489394    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50ad01a-0312-452c-997f-e928048a2c06" path="/var/lib/kubelet/pods/b50ad01a-0312-452c-997f-e928048a2c06/volumes"
Oct 29 23:06:44.896109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896075    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"]
Oct 29 23:06:44.896724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896710    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b50ad01a-0312-452c-997f-e928048a2c06" containerName="maas-api"
Oct 29 23:06:44.896787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896725    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50ad01a-0312-452c-997f-e928048a2c06" containerName="maas-api"
Oct 29 23:06:44.896787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896747    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f98991aa-3271-4b69-9b1a-191af1e3a47c" containerName="maas-api"
Oct 29 23:06:44.896787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896752    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98991aa-3271-4b69-9b1a-191af1e3a47c" containerName="maas-api"
Oct 29 23:06:44.896892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896790    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy"
Oct 29 23:06:44.896892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896799    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy"
Oct 29 23:06:44.896892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896809    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" containerName="maas-api"
Oct 29 23:06:44.896892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896816    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" containerName="maas-api"
Oct 29 23:06:44.896892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896840    4827 cpu_manager.go:414] "containerMap: removing container" podUID="96516365-f87e-456e-85dd-dcda19696285" containerName="payload-pre-processing"
Oct 29 23:06:44.896892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896844    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96516365-f87e-456e-85dd-dcda19696285" containerName="payload-pre-processing"
Oct 29 23:06:44.897160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896945    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy"
Oct 29 23:06:44.897160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896952    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0c1234-dd70-4a66-ab36-efbe8d5bf1fe" containerName="maas-api"
Oct 29 23:06:44.897160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896960    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="96516365-f87e-456e-85dd-dcda19696285" containerName="payload-pre-processing"
Oct 29 23:06:44.897160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896968    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy"
Oct 29 23:06:44.897160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896977    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50ad01a-0312-452c-997f-e928048a2c06" containerName="maas-api"
Oct 29 23:06:44.897160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.896983    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98991aa-3271-4b69-9b1a-191af1e3a47c" containerName="maas-api"
Oct 29 23:06:44.901732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.901707    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:44.926478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.926452    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-047892-serving-cert\""
Oct 29 23:06:44.931430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.931403    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"]
Oct 29 23:06:44.971905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.971868    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-metrics-certs\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:44.972089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.971914    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkntx\" (UniqueName: \"kubernetes.io/projected/59a79136-7646-4227-bac1-328e00707460-kube-api-access-dkntx\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:44.972089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.972072    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a79136-7646-4227-bac1-328e00707460-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:44.972229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:44.972159    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-maas-api-tls\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.073526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.073485    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-metrics-certs\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.073526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.073525    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkntx\" (UniqueName: \"kubernetes.io/projected/59a79136-7646-4227-bac1-328e00707460-kube-api-access-dkntx\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.073827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.073580    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a79136-7646-4227-bac1-328e00707460-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.073827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.073639    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-maas-api-tls\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.074250 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.074227    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a79136-7646-4227-bac1-328e00707460-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.076154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.076126    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-metrics-certs\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.076265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.076126    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-maas-api-tls\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.096329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.096307    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkntx\" (UniqueName: \"kubernetes.io/projected/59a79136-7646-4227-bac1-328e00707460-kube-api-access-dkntx\") pod \"maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.219837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.218049    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:45.598465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.598434    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"]
Oct 29 23:06:45.599396 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:06:45.599367    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a79136_7646_4227_bac1_328e00707460.slice/crio-53d64574230e61a89a44130330dead6fdfec5b2b5f0caaf96948b464536cc5a7 WatchSource:0}: Error finding container 53d64574230e61a89a44130330dead6fdfec5b2b5f0caaf96948b464536cc5a7: Status 404 returned error can't find the container with id 53d64574230e61a89a44130330dead6fdfec5b2b5f0caaf96948b464536cc5a7
Oct 29 23:06:45.804954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:45.804910    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx" event={"ID":"59a79136-7646-4227-bac1-328e00707460","Type":"ContainerStarted","Data":"53d64574230e61a89a44130330dead6fdfec5b2b5f0caaf96948b464536cc5a7"}
Oct 29 23:06:46.816280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:46.816241    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx" event={"ID":"59a79136-7646-4227-bac1-328e00707460","Type":"ContainerStarted","Data":"9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228"}
Oct 29 23:06:46.816745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:46.816430    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:46.859264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:46.859218    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx" podStartSLOduration=2.491526858 podStartE2EDuration="2.859206196s" totalImagesPullingTime="367.679338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:46.858003294 +0000 UTC m=+1442.480914314" watchObservedRunningTime="2026-10-29 23:06:46.859206196 +0000 UTC m=+1442.482117264"
Oct 29 23:06:48.646872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:48.646088    4827 scope.go:122] "RemoveContainer" containerID="e3441ef3f2a56bb8110bda2a1a74adff2ce5348aaf4165061cdabde34acec43a"
Oct 29 23:06:52.627230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:52.616837    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"]
Oct 29 23:06:52.627230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:52.617097    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh" podUID="5ab1abeb-5508-4146-944e-6d48e5ccbc87" containerName="maas-api" containerID="cri-o://212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5" gracePeriod=30
Oct 29 23:06:53.021011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:52.967213    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:06:54.063095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.063067    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:06:54.092796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.092738    4827 generic.go:356] "Generic (PLEG): container finished" podID="5ab1abeb-5508-4146-944e-6d48e5ccbc87" containerID="212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5" exitCode=0
Oct 29 23:06:54.093277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.093251    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh" event={"ID":"5ab1abeb-5508-4146-944e-6d48e5ccbc87","Type":"ContainerDied","Data":"212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5"}
Oct 29 23:06:54.093451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.093435    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh" event={"ID":"5ab1abeb-5508-4146-944e-6d48e5ccbc87","Type":"ContainerDied","Data":"90f92c75ab5c0fa34bfbeedbec662b14d9cfbccb57a6e61f80945b8e7f5f985f"}
Oct 29 23:06:54.093569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.093556    4827 scope.go:122] "RemoveContainer" containerID="212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5"
Oct 29 23:06:54.093867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.093849    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"
Oct 29 23:06:54.109944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.109854    4827 scope.go:122] "RemoveContainer" containerID="212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5"
Oct 29 23:06:54.110476 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:06:54.110444    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5\": container with ID starting with 212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5 not found: ID does not exist" containerID="212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5"
Oct 29 23:06:54.110630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.110484    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5"} err="failed to get container status \"212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5\": rpc error: code = NotFound desc = could not find container \"212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5\": container with ID starting with 212518662ee636e3264628192a9d85aa9b4c831f01e732aa55914acb7a210ba5 not found: ID does not exist"
Oct 29 23:06:54.149288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.149259    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-maas-api-tls\") pod \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") "
Oct 29 23:06:54.149464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.149421    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab1abeb-5508-4146-944e-6d48e5ccbc87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab1abeb-5508-4146-944e-6d48e5ccbc87-openshift-service-ca\") pod \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") "
Oct 29 23:06:54.149658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.149640    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab1abeb-5508-4146-944e-6d48e5ccbc87-kube-api-access-cj2qd\" (UniqueName: \"kubernetes.io/projected/5ab1abeb-5508-4146-944e-6d48e5ccbc87-kube-api-access-cj2qd\") pod \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") "
Oct 29 23:06:54.149782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.149748    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-metrics-certs\") pod \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\" (UID: \"5ab1abeb-5508-4146-944e-6d48e5ccbc87\") "
Oct 29 23:06:54.150092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.150068    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab1abeb-5508-4146-944e-6d48e5ccbc87-openshift-service-ca" pod "5ab1abeb-5508-4146-944e-6d48e5ccbc87" (UID: "5ab1abeb-5508-4146-944e-6d48e5ccbc87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:54.150679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.150654    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab1abeb-5508-4146-944e-6d48e5ccbc87-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:54.152089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.152060    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-maas-api-tls" pod "5ab1abeb-5508-4146-944e-6d48e5ccbc87" (UID: "5ab1abeb-5508-4146-944e-6d48e5ccbc87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:54.152177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.152153    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-metrics-certs" pod "5ab1abeb-5508-4146-944e-6d48e5ccbc87" (UID: "5ab1abeb-5508-4146-944e-6d48e5ccbc87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:54.152672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.152650    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab1abeb-5508-4146-944e-6d48e5ccbc87-kube-api-access-cj2qd" pod "5ab1abeb-5508-4146-944e-6d48e5ccbc87" (UID: "5ab1abeb-5508-4146-944e-6d48e5ccbc87"). InnerVolumeSpecName "kube-api-access-cj2qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:54.252017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.251982    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2qd\" (UniqueName: \"kubernetes.io/projected/5ab1abeb-5508-4146-944e-6d48e5ccbc87-kube-api-access-cj2qd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:54.252017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.252009    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:54.252017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.252019    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab1abeb-5508-4146-944e-6d48e5ccbc87-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:06:54.514262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.514230    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"]
Oct 29 23:06:54.528120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:54.528086    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4f2e3a-7c795447cd-8dsqh"]
Oct 29 23:06:56.490757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.490718    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab1abeb-5508-4146-944e-6d48e5ccbc87" path="/var/lib/kubelet/pods/5ab1abeb-5508-4146-944e-6d48e5ccbc87/volumes"
Oct 29 23:06:56.597503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.597467    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"]
Oct 29 23:06:56.598329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.598310    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy"
Oct 29 23:06:56.598407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.598330    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5dacfa-5542-4a7d-a809-f02354a36829" containerName="istio-proxy"
Oct 29 23:06:56.598407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.598385    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5ab1abeb-5508-4146-944e-6d48e5ccbc87" containerName="maas-api"
Oct 29 23:06:56.598407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.598394    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab1abeb-5508-4146-944e-6d48e5ccbc87" containerName="maas-api"
Oct 29 23:06:56.598563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.598552    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab1abeb-5508-4146-944e-6d48e5ccbc87" containerName="maas-api"
Oct 29 23:06:56.604223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.604165    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.627753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.627720    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"]
Oct 29 23:06:56.732419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.732376    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.732601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.732572    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.732738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.732720    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.732792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.732777    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.732861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.732846    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.732973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.732950    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wfx\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-kube-api-access-b9wfx\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.733044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.733001    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.733104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.733079    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.733159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.733123    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.834745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.834666    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.834948 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.834837    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.834978    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835023    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835076    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835111    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wfx\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-kube-api-access-b9wfx\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835152    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835193    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835214    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835230    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.835682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835646    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.837386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835801    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.837386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835958    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.837386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.835983    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.837594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.837572    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.837734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.837715    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.858309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.858278    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.858444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.858380    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wfx\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-kube-api-access-b9wfx\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:56.936803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:56.936757    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:57.284461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:57.284435    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"]
Oct 29 23:06:57.286321 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:06:57.286297    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fdb194_be85_4d72_9df4_fd9ec332ef3f.slice/crio-f643e61cb68ed9eb6f95b1772f527f1d3f6b0cb87caf0503bd6a2a3b28967d2e WatchSource:0}: Error finding container f643e61cb68ed9eb6f95b1772f527f1d3f6b0cb87caf0503bd6a2a3b28967d2e: Status 404 returned error can't find the container with id f643e61cb68ed9eb6f95b1772f527f1d3f6b0cb87caf0503bd6a2a3b28967d2e
Oct 29 23:06:57.289082 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:57.289049    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:57.289165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:57.289116    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:57.289165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:57.289146    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:06:58.139862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:58.139821    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" event={"ID":"c2fdb194-be85-4d72-9df4-fd9ec332ef3f","Type":"ContainerStarted","Data":"0d350dc35f7d683da0453be8fbf55a134d26d16875bff1498d6f68710068473f"}
Oct 29 23:06:58.140245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:58.139886    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" event={"ID":"c2fdb194-be85-4d72-9df4-fd9ec332ef3f","Type":"ContainerStarted","Data":"f643e61cb68ed9eb6f95b1772f527f1d3f6b0cb87caf0503bd6a2a3b28967d2e"}
Oct 29 23:06:58.218937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:58.218735    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podStartSLOduration=2.218709258 podStartE2EDuration="2.218709258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:58.217165262 +0000 UTC m=+1453.840076282" watchObservedRunningTime="2026-10-29 23:06:58.218709258 +0000 UTC m=+1453.841620271"
Oct 29 23:06:58.936984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:58.936954    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:06:58.938444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:58.938415    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:06:58.938588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:58.938471    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 23:06:59.304203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:59.302650    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:51788->10.134.0.194:15021: read: connection reset by peer" start-of-body=
Oct 29 23:06:59.304203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:59.302718    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:51788->10.134.0.194:15021: read: connection reset by peer"
Oct 29 23:06:59.937719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:59.937688    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:06:59.937934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:06:59.937734    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 23:07:00.155203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.155169    4827 generic.go:356] "Generic (PLEG): container finished" podID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerID="e2df71419852a05589c3650983941fc519f123096b9afb8afe52118cb159a18a" exitCode=0
Oct 29 23:07:00.155376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.155236    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" event={"ID":"739f2a27-1668-47cf-9b3e-5beab9d7cf1d","Type":"ContainerDied","Data":"e2df71419852a05589c3650983941fc519f123096b9afb8afe52118cb159a18a"}
Oct 29 23:07:00.155922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.155899    4827 scope.go:122] "RemoveContainer" containerID="e2df71419852a05589c3650983941fc519f123096b9afb8afe52118cb159a18a"
Oct 29 23:07:00.157602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.157572    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:00.157692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.157645    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:00.157692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.157683    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:00.938077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.938042    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:07:00.938496 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:00.938108    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 23:07:01.180512 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.180475    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" event={"ID":"739f2a27-1668-47cf-9b3e-5beab9d7cf1d","Type":"ContainerStarted","Data":"3ebe0e778b340f6b12dd3c7f150e18b3e10238c20b21147fdd7a1596c50e36ea"}
Oct 29 23:07:01.433973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.433935    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"]
Oct 29 23:07:01.434187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.434168    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy" containerID="cri-o://ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed" gracePeriod=30
Oct 29 23:07:01.897792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.892154    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:07:01.897792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.892322    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 29 23:07:01.897792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.892366    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 29 23:07:01.937104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.937074    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:07:01.937277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:01.937124    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 23:07:02.848367 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:02.848324    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:07:02.896790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:02.891163    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 29 23:07:02.896790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:02.891222    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 29 23:07:02.958669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:02.950555    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:07:02.958669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:02.950608    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 23:07:03.889198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:03.889157    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 29 23:07:03.889759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:03.889220    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 29 23:07:03.937216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:03.937184    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:07:03.937438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:03.937251    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 23:07:04.889326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:04.889289    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 29 23:07:04.889744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:04.889344    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 29 23:07:04.941621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:04.941594    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:07:04.942051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:04.942037    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:07:04.942754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:04.942736    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:07:05.889345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:05.889300    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 29 23:07:05.889777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:05.889369    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 29 23:07:06.759273 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.759242    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:07:06.858174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858144    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-certs\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.858366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858296    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-socket\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.858444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858396    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f062884e-7973-4797-854e-d2322a868e86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f062884e-7973-4797-854e-d2322a868e86-istio-podinfo\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.858444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858414    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-certs" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:06.858554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858461    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-envoy\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.858554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858518    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-data\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.858656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858574    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-socket" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:06.858656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858614    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-istio-token\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-istio-token\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.858820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.858799    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-credential-socket\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.859019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.859000    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-kube-api-access-plwrp\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-kube-api-access-plwrp\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.859163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.859148    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f062884e-7973-4797-854e-d2322a868e86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f062884e-7973-4797-854e-d2322a868e86-istiod-ca-cert\") pod \"f062884e-7973-4797-854e-d2322a868e86\" (UID: \"f062884e-7973-4797-854e-d2322a868e86\") "
Oct 29 23:07:06.859673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.859048    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-data" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:06.859673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.859226    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-credential-socket" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:06.859823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.859571    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f062884e-7973-4797-854e-d2322a868e86-istiod-ca-cert" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:06.860102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.860081    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.860183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.860105    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.860183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.860119    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.860183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.860135    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.860183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.860150    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f062884e-7973-4797-854e-d2322a868e86-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.862057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.862031    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-envoy" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:06.862185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.862093    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-kube-api-access-plwrp" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "kube-api-access-plwrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:06.862326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.862305    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f062884e-7973-4797-854e-d2322a868e86-istio-podinfo" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:07:06.862602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.862576    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-istio-token" pod "f062884e-7973-4797-854e-d2322a868e86" (UID: "f062884e-7973-4797-854e-d2322a868e86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:06.891509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.891193    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 29 23:07:06.891509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.891236    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 29 23:07:06.962185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.962157    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.962185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.962185    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwrp\" (UniqueName: \"kubernetes.io/projected/f062884e-7973-4797-854e-d2322a868e86-kube-api-access-plwrp\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.962378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.962195    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f062884e-7973-4797-854e-d2322a868e86-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.962378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:06.962203    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f062884e-7973-4797-854e-d2322a868e86-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:07.335499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.335461    4827 generic.go:356] "Generic (PLEG): container finished" podID="f062884e-7973-4797-854e-d2322a868e86" containerID="ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed" exitCode=0
Oct 29 23:07:07.335718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.335527    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"
Oct 29 23:07:07.335718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.335538    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" event={"ID":"f062884e-7973-4797-854e-d2322a868e86","Type":"ContainerDied","Data":"ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed"}
Oct 29 23:07:07.335718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.335597    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g" event={"ID":"f062884e-7973-4797-854e-d2322a868e86","Type":"ContainerDied","Data":"b0dff3be1f2ac19a6a8f3a5fdf002afd8144485aa5123d0a3d3366c750f29fa1"}
Oct 29 23:07:07.335718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.335619    4827 scope.go:122] "RemoveContainer" containerID="ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed"
Oct 29 23:07:07.350030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.350009    4827 scope.go:122] "RemoveContainer" containerID="299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a"
Oct 29 23:07:07.361167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.361144    4827 scope.go:122] "RemoveContainer" containerID="ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed"
Oct 29 23:07:07.361594 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:07.361565    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed\": container with ID starting with ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed not found: ID does not exist" containerID="ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed"
Oct 29 23:07:07.362187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.361603    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed"} err="failed to get container status \"ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed\": rpc error: code = NotFound desc = could not find container \"ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed\": container with ID starting with ad7b0a8360d206d6f4216ccf4d47ba894297e4f67dbfabf4db0ae4fadd653eed not found: ID does not exist"
Oct 29 23:07:07.362187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.361623    4827 scope.go:122] "RemoveContainer" containerID="299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a"
Oct 29 23:07:07.362187 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:07.361980    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a\": container with ID starting with 299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a not found: ID does not exist" containerID="299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a"
Oct 29 23:07:07.362187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.362010    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a"} err="failed to get container status \"299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a\": rpc error: code = NotFound desc = could not find container \"299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a\": container with ID starting with 299d0d42784fdb3f4b3c2aaf0e6d013235933eec91d62f06bcbb7bfd7957103a not found: ID does not exist"
Oct 29 23:07:07.400603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.400574    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"]
Oct 29 23:07:07.412593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.412571    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4f2e3a-openshift-default-7f978cb48f-5dz6g"]
Oct 29 23:07:07.893004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.892974    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:07:07.893542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.893522    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:07:07.894182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:07.894162    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:07:08.489933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:08.489899    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f062884e-7973-4797-854e-d2322a868e86" path="/var/lib/kubelet/pods/f062884e-7973-4797-854e-d2322a868e86/volumes"
Oct 29 23:07:21.788610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.788579    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"]
Oct 29 23:07:21.789409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.789389    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy"
Oct 29 23:07:21.789455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.789411    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy"
Oct 29 23:07:21.789455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.789434    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy"
Oct 29 23:07:21.789455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.789440    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy"
Oct 29 23:07:21.789545 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.789537    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy"
Oct 29 23:07:21.794230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.794212    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.812324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.812299    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"]
Oct 29 23:07:21.820318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.820297    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:07:21.820462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.820297    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:21.894017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.893985    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.894235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.894121    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.894235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.894167    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tkr\" (UniqueName: \"kubernetes.io/projected/6438ed73-8266-40ee-b4ca-e55045732844-kube-api-access-j6tkr\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.995829 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.995754    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.996002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.995904    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.996002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.995947    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tkr\" (UniqueName: \"kubernetes.io/projected/6438ed73-8266-40ee-b4ca-e55045732844-kube-api-access-j6tkr\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.996479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.996457    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:21.998223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:21.998204    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:22.020414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:22.020376    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tkr\" (UniqueName: \"kubernetes.io/projected/6438ed73-8266-40ee-b4ca-e55045732844-kube-api-access-j6tkr\") pod \"maas-api-revoke-sub-d96f14b1-mzpzc\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:22.108901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:22.108803    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:22.426151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:22.426119    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"]
Oct 29 23:07:22.426731 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:07:22.426704    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6438ed73_8266_40ee_b4ca_e55045732844.slice/crio-73f19bc6e77ce9093354b25373e1008c6be4631d3800d95d280b2ae99a315255 WatchSource:0}: Error finding container 73f19bc6e77ce9093354b25373e1008c6be4631d3800d95d280b2ae99a315255: Status 404 returned error can't find the container with id 73f19bc6e77ce9093354b25373e1008c6be4631d3800d95d280b2ae99a315255
Oct 29 23:07:22.649692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:22.649650    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc" event={"ID":"6438ed73-8266-40ee-b4ca-e55045732844","Type":"ContainerStarted","Data":"73f19bc6e77ce9093354b25373e1008c6be4631d3800d95d280b2ae99a315255"}
Oct 29 23:07:23.045388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.045318    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"]
Oct 29 23:07:23.046114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.046098    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f062884e-7973-4797-854e-d2322a868e86" containerName="istio-proxy"
Oct 29 23:07:23.049804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.049752    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.072588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.072210    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"]
Oct 29 23:07:23.143043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143016    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143106    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143224    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143315    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143388    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-kube-api-access-pv44g\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143450    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143511    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143582    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.143646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.143630    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.245583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.245553    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.245812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.245632    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.245812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.245704    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-kube-api-access-pv44g\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.245812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.245759    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.245899    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.245925    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246011    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246066    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246072    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246179    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246236    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246276    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246502    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.246555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.246525    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.248851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.248760    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.249172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.249149    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.269485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.269459    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.269654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.269632    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-kube-api-access-pv44g\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.364760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.364735    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:23.658916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.658883    4827 generic.go:356] "Generic (PLEG): container finished" podID="6438ed73-8266-40ee-b4ca-e55045732844" containerID="67301542fbfae44bd6cd7956f07c385d06adc10825844e5f9845fa2f07039f04" exitCode=0
Oct 29 23:07:23.659191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.658959    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc" event={"ID":"6438ed73-8266-40ee-b4ca-e55045732844","Type":"ContainerDied","Data":"67301542fbfae44bd6cd7956f07c385d06adc10825844e5f9845fa2f07039f04"}
Oct 29 23:07:23.683811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.683781    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"]
Oct 29 23:07:23.684946 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:07:23.684915    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb1ccfd_7044_4b45_8e8a_4da139673db6.slice/crio-66d008a207cde0e65b09199d0927e40d1e92e9e224b82a7d991e51310e26ff78 WatchSource:0}: Error finding container 66d008a207cde0e65b09199d0927e40d1e92e9e224b82a7d991e51310e26ff78: Status 404 returned error can't find the container with id 66d008a207cde0e65b09199d0927e40d1e92e9e224b82a7d991e51310e26ff78
Oct 29 23:07:23.687865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.687832    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:23.687943 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.687898    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:23.687943 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:23.687926    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:24.669872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:24.669827    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" event={"ID":"3cb1ccfd-7044-4b45-8e8a-4da139673db6","Type":"ContainerStarted","Data":"8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01"}
Oct 29 23:07:24.670294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:24.669903    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" event={"ID":"3cb1ccfd-7044-4b45-8e8a-4da139673db6","Type":"ContainerStarted","Data":"66d008a207cde0e65b09199d0927e40d1e92e9e224b82a7d991e51310e26ff78"}
Oct 29 23:07:24.744863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:24.734799    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podStartSLOduration=1.734762043 podStartE2EDuration="1.734762043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:24.716435995 +0000 UTC m=+1480.339347049" watchObservedRunningTime="2026-10-29 23:07:24.734762043 +0000 UTC m=+1480.357686162"
Oct 29 23:07:25.049801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.037300    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:25.147802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.145808    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6438ed73-8266-40ee-b4ca-e55045732844-kube-api-access-j6tkr\" (UniqueName: \"kubernetes.io/projected/6438ed73-8266-40ee-b4ca-e55045732844-kube-api-access-j6tkr\") pod \"6438ed73-8266-40ee-b4ca-e55045732844\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") "
Oct 29 23:07:25.147802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.146160    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-xks-ca\") pod \"6438ed73-8266-40ee-b4ca-e55045732844\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") "
Oct 29 23:07:25.147802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.146223    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-openshift-ca\") pod \"6438ed73-8266-40ee-b4ca-e55045732844\" (UID: \"6438ed73-8266-40ee-b4ca-e55045732844\") "
Oct 29 23:07:25.147802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.147317    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-openshift-ca" pod "6438ed73-8266-40ee-b4ca-e55045732844" (UID: "6438ed73-8266-40ee-b4ca-e55045732844"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:25.250028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.248303    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:25.297502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.297462    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6438ed73-8266-40ee-b4ca-e55045732844-kube-api-access-j6tkr" pod "6438ed73-8266-40ee-b4ca-e55045732844" (UID: "6438ed73-8266-40ee-b4ca-e55045732844"). InnerVolumeSpecName "kube-api-access-j6tkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:25.297879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.297834    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-xks-ca" pod "6438ed73-8266-40ee-b4ca-e55045732844" (UID: "6438ed73-8266-40ee-b4ca-e55045732844"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:25.349424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.349378    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6438ed73-8266-40ee-b4ca-e55045732844-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:25.349424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.349413    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6tkr\" (UniqueName: \"kubernetes.io/projected/6438ed73-8266-40ee-b4ca-e55045732844-kube-api-access-j6tkr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:25.371239 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.371168    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:25.381162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.381128    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 23:07:25.381353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.381190    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 23:07:25.699449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.699409    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc" event={"ID":"6438ed73-8266-40ee-b4ca-e55045732844","Type":"ContainerDied","Data":"73f19bc6e77ce9093354b25373e1008c6be4631d3800d95d280b2ae99a315255"}
Oct 29 23:07:25.699862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.699487    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f19bc6e77ce9093354b25373e1008c6be4631d3800d95d280b2ae99a315255"
Oct 29 23:07:25.699862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:25.699559    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"
Oct 29 23:07:26.381789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.370069    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 23:07:26.381789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.370122    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 23:07:26.692883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.692752    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"]
Oct 29 23:07:26.693868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.693835    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6438ed73-8266-40ee-b4ca-e55045732844" containerName="revoke-keys"
Oct 29 23:07:26.693868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.693860    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6438ed73-8266-40ee-b4ca-e55045732844" containerName="revoke-keys"
Oct 29 23:07:26.694085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.694069    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6438ed73-8266-40ee-b4ca-e55045732844" containerName="revoke-keys"
Oct 29 23:07:26.698176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.698158    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.715147 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.715115    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"]
Oct 29 23:07:26.791987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.791944    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792114    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792162    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792215    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vk5\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-kube-api-access-h5vk5\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792263    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792293    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792359    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.792638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.792411    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.907151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.907054    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914000    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vk5\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-kube-api-access-h5vk5\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914145    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914221    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914381    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914481    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914581    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914654    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.914814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.914738    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.915395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.907809    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.915888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.915861    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.918615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.918588    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.918807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.918663    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.919702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.919645    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.928786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.925488    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.973680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.966652    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.999807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.968922    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:26.999807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:26.995976    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vk5\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-kube-api-access-h5vk5\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:27.030802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.028248    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:27.377793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.376794    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 23:07:27.377793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.376857    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 23:07:27.413683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.413657    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"]
Oct 29 23:07:27.415230 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:07:27.415207    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b649c51_546f_4ad2_b36d_ddd215dcb8f7.slice/crio-b2006f2003580ee0d8476d68f9382ecc86c38e1260ff4656737bbf168da4ea50 WatchSource:0}: Error finding container b2006f2003580ee0d8476d68f9382ecc86c38e1260ff4656737bbf168da4ea50: Status 404 returned error can't find the container with id b2006f2003580ee0d8476d68f9382ecc86c38e1260ff4656737bbf168da4ea50
Oct 29 23:07:27.418119 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.418087    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:27.418212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.418156    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:27.421331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.418601    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:07:27.724653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.724563    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" event={"ID":"6b649c51-546f-4ad2-b36d-ddd215dcb8f7","Type":"ContainerStarted","Data":"c28f89e53885e6d18f93c9d00a0dc5b0b0dee057a9d307cbf719fad352c0ec30"}
Oct 29 23:07:27.725158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:27.724662    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" event={"ID":"6b649c51-546f-4ad2-b36d-ddd215dcb8f7","Type":"ContainerStarted","Data":"b2006f2003580ee0d8476d68f9382ecc86c38e1260ff4656737bbf168da4ea50"}
Oct 29 23:07:28.029495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:28.029411    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:28.030899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:28.030871    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 29 23:07:28.031043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:28.030927    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 29 23:07:28.371282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:28.371174    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 23:07:28.371542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:28.371513    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 23:07:29.028926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:29.028890    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 29 23:07:29.029427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:29.028951    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 29 23:07:29.384855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:29.371060    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 23:07:29.384855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:29.371114    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 23:07:29.407791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:29.401282    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podStartSLOduration=3.401260423 podStartE2EDuration="3.401260423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:27.792130154 +0000 UTC m=+1483.415041195" watchObservedRunningTime="2026-10-29 23:07:29.401260423 +0000 UTC m=+1485.024171444"
Oct 29 23:07:29.407791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:29.407507    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"]
Oct 29 23:07:30.031864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:30.031828    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 29 23:07:30.053956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:30.053912    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 29 23:07:30.392093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:30.392063    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 23:07:30.418982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:30.418938    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 23:07:31.028885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:31.028852    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 29 23:07:31.029047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:31.028900    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 29 23:07:31.369182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:31.369090    4827 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 23:07:31.392974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:31.392925    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 23:07:32.030169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:32.030134    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 29 23:07:32.030442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:32.030411    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 29 23:07:32.636895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:32.636864    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:32.637383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:32.637287    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:32.638294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:32.638274    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:07:33.025324 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:33.025227    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:07:33.059669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:33.059605    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 29 23:07:33.078894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:33.078855    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 29 23:07:33.192790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:33.192312    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"]
Oct 29 23:07:33.192790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:33.192558    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" containerID="cri-o://cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0" gracePeriod=30
Oct 29 23:07:34.029085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.029051    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 29 23:07:34.029459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.029105    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 29 23:07:34.408495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.408430    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv" podUID="9d12ed58-e87f-4c42-9cdb-5c683354197f" containerName="payload-processing" containerID="cri-o://0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f" gracePeriod=25
Oct 29 23:07:34.704022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.703983    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"]
Oct 29 23:07:34.709459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.709438    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.724878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.724849    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"]
Oct 29 23:07:34.731869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.731845    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:34.731869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.731868    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:07:34.737944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.737922    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:07:34.849453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.843945    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d12ed58-e87f-4c42-9cdb-5c683354197f-kube-api-access-2b78c\" (UniqueName: \"kubernetes.io/projected/9d12ed58-e87f-4c42-9cdb-5c683354197f-kube-api-access-2b78c\") pod \"9d12ed58-e87f-4c42-9cdb-5c683354197f\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") "
Oct 29 23:07:34.849453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.844160    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d12ed58-e87f-4c42-9cdb-5c683354197f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d12ed58-e87f-4c42-9cdb-5c683354197f-plugins-config-volume\") pod \"9d12ed58-e87f-4c42-9cdb-5c683354197f\" (UID: \"9d12ed58-e87f-4c42-9cdb-5c683354197f\") "
Oct 29 23:07:34.849453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.844782    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zql2t\" (UniqueName: \"kubernetes.io/projected/e563d1c5-3de7-496f-8259-82b20c410a9f-kube-api-access-zql2t\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.849453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.844940    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.849453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.845073    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.861134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.861100    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d12ed58-e87f-4c42-9cdb-5c683354197f-plugins-config-volume" pod "9d12ed58-e87f-4c42-9cdb-5c683354197f" (UID: "9d12ed58-e87f-4c42-9cdb-5c683354197f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:34.873725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.872349    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d12ed58-e87f-4c42-9cdb-5c683354197f-kube-api-access-2b78c" pod "9d12ed58-e87f-4c42-9cdb-5c683354197f" (UID: "9d12ed58-e87f-4c42-9cdb-5c683354197f"). InnerVolumeSpecName "kube-api-access-2b78c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:34.946737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.946695    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.946971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.946951    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zql2t\" (UniqueName: \"kubernetes.io/projected/e563d1c5-3de7-496f-8259-82b20c410a9f-kube-api-access-zql2t\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.947105 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.947088    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.947242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.947226    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b78c\" (UniqueName: \"kubernetes.io/projected/9d12ed58-e87f-4c42-9cdb-5c683354197f-kube-api-access-2b78c\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:34.947314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.947247    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d12ed58-e87f-4c42-9cdb-5c683354197f-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:34.948084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.948058    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.956844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.955990    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:34.970341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:34.970315    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zql2t\" (UniqueName: \"kubernetes.io/projected/e563d1c5-3de7-496f-8259-82b20c410a9f-kube-api-access-zql2t\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:35.034020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.033984    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:35.034462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.034442    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:35.035145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.035127    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:07:35.057749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.057727    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:35.104487 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.104454    4827 generic.go:356] "Generic (PLEG): container finished" podID="9d12ed58-e87f-4c42-9cdb-5c683354197f" containerID="0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f" exitCode=0
Oct 29 23:07:35.104678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.104510    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"
Oct 29 23:07:35.104678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.104537    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv" event={"ID":"9d12ed58-e87f-4c42-9cdb-5c683354197f","Type":"ContainerDied","Data":"0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f"}
Oct 29 23:07:35.104678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.104587    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv" event={"ID":"9d12ed58-e87f-4c42-9cdb-5c683354197f","Type":"ContainerDied","Data":"857785eeb702dbc502440a0c4090442163faa69f4c9f118a8e2e9bffcd2d099a"}
Oct 29 23:07:35.104678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.104615    4827 scope.go:122] "RemoveContainer" containerID="0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f"
Oct 29 23:07:35.117012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.116993    4827 scope.go:122] "RemoveContainer" containerID="0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f"
Oct 29 23:07:35.117304 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:35.117286    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f\": container with ID starting with 0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f not found: ID does not exist" containerID="0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f"
Oct 29 23:07:35.117373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.117310    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f"} err="failed to get container status \"0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f\": rpc error: code = NotFound desc = could not find container \"0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f\": container with ID starting with 0c86c41ddb6d1158198e59d2289017d7491a89e7437a0da1dda441e87dcb695f not found: ID does not exist"
Oct 29 23:07:35.185899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.185850    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"]
Oct 29 23:07:35.190554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.190527    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f7d4b-566d66d6b4-d2vjv"]
Oct 29 23:07:35.381528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:35.381499    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"]
Oct 29 23:07:35.382336 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:07:35.382315    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode563d1c5_3de7_496f_8259_82b20c410a9f.slice/crio-ca7951f2b5c37015cd7f99a68849096aa97d10619d128077828e94ff4669ef99 WatchSource:0}: Error finding container ca7951f2b5c37015cd7f99a68849096aa97d10619d128077828e94ff4669ef99: Status 404 returned error can't find the container with id ca7951f2b5c37015cd7f99a68849096aa97d10619d128077828e94ff4669ef99
Oct 29 23:07:36.114442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:36.114326    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7" event={"ID":"e563d1c5-3de7-496f-8259-82b20c410a9f","Type":"ContainerStarted","Data":"06383392850a123bb01d7f6763974df188025320df8c266b1105e150414a059a"}
Oct 29 23:07:36.114442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:36.114395    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7" event={"ID":"e563d1c5-3de7-496f-8259-82b20c410a9f","Type":"ContainerStarted","Data":"ca7951f2b5c37015cd7f99a68849096aa97d10619d128077828e94ff4669ef99"}
Oct 29 23:07:36.157538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:36.157488    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7" podStartSLOduration=2.157474158 podStartE2EDuration="2.157474158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:36.153919331 +0000 UTC m=+1491.776830341" watchObservedRunningTime="2026-10-29 23:07:36.157474158 +0000 UTC m=+1491.780385195"
Oct 29 23:07:36.489954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:36.489916    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d12ed58-e87f-4c42-9cdb-5c683354197f" path="/var/lib/kubelet/pods/9d12ed58-e87f-4c42-9cdb-5c683354197f/volumes"
Oct 29 23:07:37.123494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:37.123453    4827 generic.go:356] "Generic (PLEG): container finished" podID="e563d1c5-3de7-496f-8259-82b20c410a9f" containerID="06383392850a123bb01d7f6763974df188025320df8c266b1105e150414a059a" exitCode=0
Oct 29 23:07:37.123939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:37.123512    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7" event={"ID":"e563d1c5-3de7-496f-8259-82b20c410a9f","Type":"ContainerDied","Data":"06383392850a123bb01d7f6763974df188025320df8c266b1105e150414a059a"}
Oct 29 23:07:37.287231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:37.287201    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:07:37.287403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:37.287265    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:07:38.434906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.429044    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:38.528455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.528424    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-openshift-ca\") pod \"e563d1c5-3de7-496f-8259-82b20c410a9f\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") "
Oct 29 23:07:38.528654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.528626    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-xks-ca\") pod \"e563d1c5-3de7-496f-8259-82b20c410a9f\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") "
Oct 29 23:07:38.528747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.528690    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e563d1c5-3de7-496f-8259-82b20c410a9f-kube-api-access-zql2t\" (UniqueName: \"kubernetes.io/projected/e563d1c5-3de7-496f-8259-82b20c410a9f-kube-api-access-zql2t\") pod \"e563d1c5-3de7-496f-8259-82b20c410a9f\" (UID: \"e563d1c5-3de7-496f-8259-82b20c410a9f\") "
Oct 29 23:07:38.530352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.530323    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-openshift-ca" pod "e563d1c5-3de7-496f-8259-82b20c410a9f" (UID: "e563d1c5-3de7-496f-8259-82b20c410a9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:38.532349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.532322    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e563d1c5-3de7-496f-8259-82b20c410a9f-kube-api-access-zql2t" pod "e563d1c5-3de7-496f-8259-82b20c410a9f" (UID: "e563d1c5-3de7-496f-8259-82b20c410a9f"). InnerVolumeSpecName "kube-api-access-zql2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:38.534817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.534463    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-xks-ca" pod "e563d1c5-3de7-496f-8259-82b20c410a9f" (UID: "e563d1c5-3de7-496f-8259-82b20c410a9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:38.608485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.608457    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"]
Oct 29 23:07:38.609297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.609276    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9d12ed58-e87f-4c42-9cdb-5c683354197f" containerName="payload-processing"
Oct 29 23:07:38.609297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.609296    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d12ed58-e87f-4c42-9cdb-5c683354197f" containerName="payload-processing"
Oct 29 23:07:38.609450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.609341    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e563d1c5-3de7-496f-8259-82b20c410a9f" containerName="revoke-keys"
Oct 29 23:07:38.609450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.609346    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e563d1c5-3de7-496f-8259-82b20c410a9f" containerName="revoke-keys"
Oct 29 23:07:38.609450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.609440    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e563d1c5-3de7-496f-8259-82b20c410a9f" containerName="revoke-keys"
Oct 29 23:07:38.609558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.609454    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d12ed58-e87f-4c42-9cdb-5c683354197f" containerName="payload-processing"
Oct 29 23:07:38.613598 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.613576    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.617013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.616993    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:07:38.633061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633025    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"]
Oct 29 23:07:38.633183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633068    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-certs\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633162    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-token\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633257    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-envoy\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633304    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-podinfo\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633347    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-data\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633457    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-kube-api-access-k4sh5\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-kube-api-access-k4sh5\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633537    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-socket\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633572    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-credential-socket\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633645    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd3d0b2-dc85-4627-a891-0c5d33639748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3d0b2-dc85-4627-a891-0c5d33639748-istiod-ca-cert\") pod \"bbd3d0b2-dc85-4627-a891-0c5d33639748\" (UID: \"bbd3d0b2-dc85-4627-a891-0c5d33639748\") "
Oct 29 23:07:38.633747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.633713    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-certs" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:38.634242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.634210    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.634329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.634280    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4kl\" (UniqueName: \"kubernetes.io/projected/c1581006-f0c0-49e7-bff0-bc00ef834dd0-kube-api-access-rm4kl\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.634494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.634470    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.635361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.634861    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.635361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.634885    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.635361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.634898    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e563d1c5-3de7-496f-8259-82b20c410a9f-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.635361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.634916    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zql2t\" (UniqueName: \"kubernetes.io/projected/e563d1c5-3de7-496f-8259-82b20c410a9f-kube-api-access-zql2t\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.635963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.635851    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-socket" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:38.636334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.636289    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-credential-socket" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:38.636500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.636441    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-kube-api-access-k4sh5" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "kube-api-access-k4sh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:38.636742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.636705    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-data" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:38.636868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.636783    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd3d0b2-dc85-4627-a891-0c5d33639748-istiod-ca-cert" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:38.637288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.637255    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-envoy" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:38.637381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.637336    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-podinfo" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:07:38.642527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.642503    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-token" pod "bbd3d0b2-dc85-4627-a891-0c5d33639748" (UID: "bbd3d0b2-dc85-4627-a891-0c5d33639748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:38.737213 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.737114    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.737213 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.737169    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4kl\" (UniqueName: \"kubernetes.io/projected/c1581006-f0c0-49e7-bff0-bc00ef834dd0-kube-api-access-rm4kl\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.737458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.737338    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.738168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738139    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.738368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738348    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.738439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738374    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.738439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738389    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.738439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738400    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.738439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738413    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4sh5\" (UniqueName: \"kubernetes.io/projected/bbd3d0b2-dc85-4627-a891-0c5d33639748-kube-api-access-k4sh5\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.738439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738427    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.738719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738440    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3d0b2-dc85-4627-a891-0c5d33639748-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.738719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.738454    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3d0b2-dc85-4627-a891-0c5d33639748-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:38.739547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.739523    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.760921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.760891    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4kl\" (UniqueName: \"kubernetes.io/projected/c1581006-f0c0-49e7-bff0-bc00ef834dd0-kube-api-access-rm4kl\") pod \"maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:38.944904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:38.944872    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:39.138678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.138653    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"
Oct 29 23:07:39.138897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.138686    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7" event={"ID":"e563d1c5-3de7-496f-8259-82b20c410a9f","Type":"ContainerDied","Data":"ca7951f2b5c37015cd7f99a68849096aa97d10619d128077828e94ff4669ef99"}
Oct 29 23:07:39.138897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.138734    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca7951f2b5c37015cd7f99a68849096aa97d10619d128077828e94ff4669ef99"
Oct 29 23:07:39.140550 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.140521    4827 generic.go:356] "Generic (PLEG): container finished" podID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerID="cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0" exitCode=0
Oct 29 23:07:39.140691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.140569    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"
Oct 29 23:07:39.140691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.140642    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" event={"ID":"bbd3d0b2-dc85-4627-a891-0c5d33639748","Type":"ContainerDied","Data":"cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0"}
Oct 29 23:07:39.140691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.140686    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2" event={"ID":"bbd3d0b2-dc85-4627-a891-0c5d33639748","Type":"ContainerDied","Data":"d685149585e990cbaf880c4b8ffff8efcaba5cdef76f179368008698cc07d9c0"}
Oct 29 23:07:39.141013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.140720    4827 scope.go:122] "RemoveContainer" containerID="cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0"
Oct 29 23:07:39.153334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.153318    4827 scope.go:122] "RemoveContainer" containerID="56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118"
Oct 29 23:07:39.162834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.162816    4827 scope.go:122] "RemoveContainer" containerID="cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0"
Oct 29 23:07:39.163169 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:39.163105    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0\": container with ID starting with cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0 not found: ID does not exist" containerID="cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0"
Oct 29 23:07:39.163169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.163133    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0"} err="failed to get container status \"cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0\": rpc error: code = NotFound desc = could not find container \"cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0\": container with ID starting with cc9d88acfe1427bbd25d838f583eac07ebfff8891f850384c8970d5c96cda7a0 not found: ID does not exist"
Oct 29 23:07:39.163169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.163149    4827 scope.go:122] "RemoveContainer" containerID="56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118"
Oct 29 23:07:39.163451 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:39.163431    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118\": container with ID starting with 56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118 not found: ID does not exist" containerID="56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118"
Oct 29 23:07:39.163509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.163458    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118"} err="failed to get container status \"56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118\": rpc error: code = NotFound desc = could not find container \"56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118\": container with ID starting with 56fb59ced8e660c2a24f522734e33b63410683c36c68a4e8754f56f5f0892118 not found: ID does not exist"
Oct 29 23:07:39.214029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.214004    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"]
Oct 29 23:07:39.217239 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.217218    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d18f7-openshift-default-64967dd9bf-w68n2"]
Oct 29 23:07:39.256674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:39.256617    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"]
Oct 29 23:07:39.257352 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:07:39.257327    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1581006_f0c0_49e7_bff0_bc00ef834dd0.slice/crio-81319d5fa599e29515873f25cb4c838aca3da67e1bed75e79be5f45f86ad3f82 WatchSource:0}: Error finding container 81319d5fa599e29515873f25cb4c838aca3da67e1bed75e79be5f45f86ad3f82: Status 404 returned error can't find the container with id 81319d5fa599e29515873f25cb4c838aca3da67e1bed75e79be5f45f86ad3f82
Oct 29 23:07:40.175401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:40.175354    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4" event={"ID":"c1581006-f0c0-49e7-bff0-bc00ef834dd0","Type":"ContainerStarted","Data":"37bca9cbe4e8efb5b301a6b2c604eb6a824661cad373eb1ffea764bb156c2ec0"}
Oct 29 23:07:40.175909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:40.175419    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4" event={"ID":"c1581006-f0c0-49e7-bff0-bc00ef834dd0","Type":"ContainerStarted","Data":"81319d5fa599e29515873f25cb4c838aca3da67e1bed75e79be5f45f86ad3f82"}
Oct 29 23:07:40.219387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:40.219342    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4" podStartSLOduration=2.219328098 podStartE2EDuration="2.219328098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:40.215407286 +0000 UTC m=+1495.838318305" watchObservedRunningTime="2026-10-29 23:07:40.219328098 +0000 UTC m=+1495.842239163"
Oct 29 23:07:40.489585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:40.489513    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" path="/var/lib/kubelet/pods/bbd3d0b2-dc85-4627-a891-0c5d33639748/volumes"
Oct 29 23:07:41.187241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:41.187202    4827 generic.go:356] "Generic (PLEG): container finished" podID="c1581006-f0c0-49e7-bff0-bc00ef834dd0" containerID="37bca9cbe4e8efb5b301a6b2c604eb6a824661cad373eb1ffea764bb156c2ec0" exitCode=0
Oct 29 23:07:41.187646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:41.187274    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4" event={"ID":"c1581006-f0c0-49e7-bff0-bc00ef834dd0","Type":"ContainerDied","Data":"37bca9cbe4e8efb5b301a6b2c604eb6a824661cad373eb1ffea764bb156c2ec0"}
Oct 29 23:07:42.369891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.369864    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:42.404611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.404583    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-openshift-ca\") pod \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") "
Oct 29 23:07:42.404807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.404720    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1581006-f0c0-49e7-bff0-bc00ef834dd0-kube-api-access-rm4kl\" (UniqueName: \"kubernetes.io/projected/c1581006-f0c0-49e7-bff0-bc00ef834dd0-kube-api-access-rm4kl\") pod \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") "
Oct 29 23:07:42.404921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.404908    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-xks-ca\") pod \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\" (UID: \"c1581006-f0c0-49e7-bff0-bc00ef834dd0\") "
Oct 29 23:07:42.405013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.404991    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-openshift-ca" pod "c1581006-f0c0-49e7-bff0-bc00ef834dd0" (UID: "c1581006-f0c0-49e7-bff0-bc00ef834dd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:42.405349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.405329    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:42.406644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.406623    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1581006-f0c0-49e7-bff0-bc00ef834dd0-kube-api-access-rm4kl" pod "c1581006-f0c0-49e7-bff0-bc00ef834dd0" (UID: "c1581006-f0c0-49e7-bff0-bc00ef834dd0"). InnerVolumeSpecName "kube-api-access-rm4kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:42.406880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.406854    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-xks-ca" pod "c1581006-f0c0-49e7-bff0-bc00ef834dd0" (UID: "c1581006-f0c0-49e7-bff0-bc00ef834dd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:42.506558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.506480    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1581006-f0c0-49e7-bff0-bc00ef834dd0-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:42.506558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.506522    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4kl\" (UniqueName: \"kubernetes.io/projected/c1581006-f0c0-49e7-bff0-bc00ef834dd0-kube-api-access-rm4kl\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:42.693554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.693528    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"]
Oct 29 23:07:42.694207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.694192    4827 cpu_manager.go:414] "containerMap: removing container" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy"
Oct 29 23:07:42.694281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.694208    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy"
Oct 29 23:07:42.694281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.694243    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c1581006-f0c0-49e7-bff0-bc00ef834dd0" containerName="revoke-keys"
Oct 29 23:07:42.694281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.694249    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1581006-f0c0-49e7-bff0-bc00ef834dd0" containerName="revoke-keys"
Oct 29 23:07:42.694374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.694342    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy"
Oct 29 23:07:42.694374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.694356    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1581006-f0c0-49e7-bff0-bc00ef834dd0" containerName="revoke-keys"
Oct 29 23:07:42.697958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.697941    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.717078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.717050    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"]
Oct 29 23:07:42.810421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.810330    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6tj\" (UniqueName: \"kubernetes.io/projected/ce5363d4-9c46-4624-87a5-d1dae17e1b19-kube-api-access-9t6tj\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.810577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.810514    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.810672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.810644    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.912318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.912286    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.912476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.912370    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6tj\" (UniqueName: \"kubernetes.io/projected/ce5363d4-9c46-4624-87a5-d1dae17e1b19-kube-api-access-9t6tj\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.912576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.912559    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.913046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.913022    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.914913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.914889    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:42.935299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:42.935276    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6tj\" (UniqueName: \"kubernetes.io/projected/ce5363d4-9c46-4624-87a5-d1dae17e1b19-kube-api-access-9t6tj\") pod \"maas-api-revoke-sub-823a66e9-2cdjj\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:43.012037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:43.012008    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:43.202256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:43.202230    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"
Oct 29 23:07:43.202430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:43.202261    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4" event={"ID":"c1581006-f0c0-49e7-bff0-bc00ef834dd0","Type":"ContainerDied","Data":"81319d5fa599e29515873f25cb4c838aca3da67e1bed75e79be5f45f86ad3f82"}
Oct 29 23:07:43.202430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:43.202324    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81319d5fa599e29515873f25cb4c838aca3da67e1bed75e79be5f45f86ad3f82"
Oct 29 23:07:43.320026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:43.319962    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"]
Oct 29 23:07:43.320135 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:07:43.320044    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5363d4_9c46_4624_87a5_d1dae17e1b19.slice/crio-daae4de16a1203320ee8c3b620ac0b847c0d223dcefb56015c33a997d369693b WatchSource:0}: Error finding container daae4de16a1203320ee8c3b620ac0b847c0d223dcefb56015c33a997d369693b: Status 404 returned error can't find the container with id daae4de16a1203320ee8c3b620ac0b847c0d223dcefb56015c33a997d369693b
Oct 29 23:07:44.212253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:44.212218    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj" event={"ID":"ce5363d4-9c46-4624-87a5-d1dae17e1b19","Type":"ContainerStarted","Data":"18c537f68811d28a0fb8ebee80ce7d1b317ab0b6974702d6e12e6271d270e122"}
Oct 29 23:07:44.212685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:44.212273    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj" event={"ID":"ce5363d4-9c46-4624-87a5-d1dae17e1b19","Type":"ContainerStarted","Data":"daae4de16a1203320ee8c3b620ac0b847c0d223dcefb56015c33a997d369693b"}
Oct 29 23:07:44.254606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:44.254552    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj" podStartSLOduration=2.254538326 podStartE2EDuration="2.254538326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:44.251834549 +0000 UTC m=+1499.874745590" watchObservedRunningTime="2026-10-29 23:07:44.254538326 +0000 UTC m=+1499.877449345"
Oct 29 23:07:44.982624 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:44.982597    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:07:44.982959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:44.982939    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:07:45.232308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:45.232264    4827 generic.go:356] "Generic (PLEG): container finished" podID="ce5363d4-9c46-4624-87a5-d1dae17e1b19" containerID="18c537f68811d28a0fb8ebee80ce7d1b317ab0b6974702d6e12e6271d270e122" exitCode=0
Oct 29 23:07:45.232756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:45.232347    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj" event={"ID":"ce5363d4-9c46-4624-87a5-d1dae17e1b19","Type":"ContainerDied","Data":"18c537f68811d28a0fb8ebee80ce7d1b317ab0b6974702d6e12e6271d270e122"}
Oct 29 23:07:46.407621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.407593    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:46.584520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.584428    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-openshift-ca\") pod \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") "
Oct 29 23:07:46.584520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.584507    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5363d4-9c46-4624-87a5-d1dae17e1b19-kube-api-access-9t6tj\" (UniqueName: \"kubernetes.io/projected/ce5363d4-9c46-4624-87a5-d1dae17e1b19-kube-api-access-9t6tj\") pod \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") "
Oct 29 23:07:46.584806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.584749    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-xks-ca\") pod \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\" (UID: \"ce5363d4-9c46-4624-87a5-d1dae17e1b19\") "
Oct 29 23:07:46.584930 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.584906    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-openshift-ca" pod "ce5363d4-9c46-4624-87a5-d1dae17e1b19" (UID: "ce5363d4-9c46-4624-87a5-d1dae17e1b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:46.585867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.585841    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:46.586849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.586822    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-xks-ca" pod "ce5363d4-9c46-4624-87a5-d1dae17e1b19" (UID: "ce5363d4-9c46-4624-87a5-d1dae17e1b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:46.586943 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.586867    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5363d4-9c46-4624-87a5-d1dae17e1b19-kube-api-access-9t6tj" pod "ce5363d4-9c46-4624-87a5-d1dae17e1b19" (UID: "ce5363d4-9c46-4624-87a5-d1dae17e1b19"). InnerVolumeSpecName "kube-api-access-9t6tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:46.688208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.688167    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6tj\" (UniqueName: \"kubernetes.io/projected/ce5363d4-9c46-4624-87a5-d1dae17e1b19-kube-api-access-9t6tj\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:46.688208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:46.688211    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5363d4-9c46-4624-87a5-d1dae17e1b19-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:47.248071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:47.248036    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"
Oct 29 23:07:47.248071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:47.248062    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj" event={"ID":"ce5363d4-9c46-4624-87a5-d1dae17e1b19","Type":"ContainerDied","Data":"daae4de16a1203320ee8c3b620ac0b847c0d223dcefb56015c33a997d369693b"}
Oct 29 23:07:47.248291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:47.248114    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daae4de16a1203320ee8c3b620ac0b847c0d223dcefb56015c33a997d369693b"
Oct 29 23:07:48.285741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:48.285704    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"]
Oct 29 23:07:48.286212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:48.285943    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp" podUID="9d375c9d-14dd-4f68-920d-972e801e909f" containerName="maas-api" containerID="cri-o://c0409875da5ae2c50f4b3eb44d2ab938ca91cef001f3d61d6e421d7ea074baac" gracePeriod=30
Oct 29 23:07:49.121344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.120329    4827 scope.go:122] "RemoveContainer" containerID="f51377169660622a7bc4dc7ceaf3eb5380153f7c0222886553dc5c201f7fc4ac"
Oct 29 23:07:49.179818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.179791    4827 scope.go:122] "RemoveContainer" containerID="72a2ada16a4d90ec8fbef5ca9b1a882e9e3d98a6083aa8549e74240e3b96eabb"
Oct 29 23:07:49.192347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.192325    4827 scope.go:122] "RemoveContainer" containerID="fccd26e6130785dc25de0c4d2c7a30a343168952f84721bf78376534c0ba67c8"
Oct 29 23:07:49.317098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.317052    4827 generic.go:356] "Generic (PLEG): container finished" podID="9d375c9d-14dd-4f68-920d-972e801e909f" containerID="c0409875da5ae2c50f4b3eb44d2ab938ca91cef001f3d61d6e421d7ea074baac" exitCode=0
Oct 29 23:07:49.317392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.317114    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp" event={"ID":"9d375c9d-14dd-4f68-920d-972e801e909f","Type":"ContainerDied","Data":"c0409875da5ae2c50f4b3eb44d2ab938ca91cef001f3d61d6e421d7ea074baac"}
Oct 29 23:07:49.322354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.322334    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:07:49.326641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.326619    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-maas-api-tls\") pod \"9d375c9d-14dd-4f68-920d-972e801e909f\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") "
Oct 29 23:07:49.326730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.326715    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-metrics-certs\") pod \"9d375c9d-14dd-4f68-920d-972e801e909f\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") "
Oct 29 23:07:49.326974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.326953    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d375c9d-14dd-4f68-920d-972e801e909f-kube-api-access-j9f86\" (UniqueName: \"kubernetes.io/projected/9d375c9d-14dd-4f68-920d-972e801e909f-kube-api-access-j9f86\") pod \"9d375c9d-14dd-4f68-920d-972e801e909f\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") "
Oct 29 23:07:49.327107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.327090    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d375c9d-14dd-4f68-920d-972e801e909f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d375c9d-14dd-4f68-920d-972e801e909f-openshift-service-ca\") pod \"9d375c9d-14dd-4f68-920d-972e801e909f\" (UID: \"9d375c9d-14dd-4f68-920d-972e801e909f\") "
Oct 29 23:07:49.327742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.327680    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d375c9d-14dd-4f68-920d-972e801e909f-openshift-service-ca" pod "9d375c9d-14dd-4f68-920d-972e801e909f" (UID: "9d375c9d-14dd-4f68-920d-972e801e909f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:49.328176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.327884    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d375c9d-14dd-4f68-920d-972e801e909f-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:49.329340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.329299    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-metrics-certs" pod "9d375c9d-14dd-4f68-920d-972e801e909f" (UID: "9d375c9d-14dd-4f68-920d-972e801e909f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:49.330042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.329992    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-maas-api-tls" pod "9d375c9d-14dd-4f68-920d-972e801e909f" (UID: "9d375c9d-14dd-4f68-920d-972e801e909f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:49.330042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.330007    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d375c9d-14dd-4f68-920d-972e801e909f-kube-api-access-j9f86" pod "9d375c9d-14dd-4f68-920d-972e801e909f" (UID: "9d375c9d-14dd-4f68-920d-972e801e909f"). InnerVolumeSpecName "kube-api-access-j9f86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:49.429473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.429437    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:49.429473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.429473    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d375c9d-14dd-4f68-920d-972e801e909f-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:49.429675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:49.429488    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9f86\" (UniqueName: \"kubernetes.io/projected/9d375c9d-14dd-4f68-920d-972e801e909f-kube-api-access-j9f86\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.325571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:50.325544    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"
Oct 29 23:07:50.326033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:50.325583    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp" event={"ID":"9d375c9d-14dd-4f68-920d-972e801e909f","Type":"ContainerDied","Data":"1e53d2c2427bec986ca8fd2560a9cabc7566bbdf7897f90015c31777b7282a95"}
Oct 29 23:07:50.326033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:50.325650    4827 scope.go:122] "RemoveContainer" containerID="c0409875da5ae2c50f4b3eb44d2ab938ca91cef001f3d61d6e421d7ea074baac"
Oct 29 23:07:50.378888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:50.378858    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"]
Oct 29 23:07:50.391404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:50.391379    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d67deb-66cf9998fb-cn9wp"]
Oct 29 23:07:50.490604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:50.490571    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d375c9d-14dd-4f68-920d-972e801e909f" path="/var/lib/kubelet/pods/9d375c9d-14dd-4f68-920d-972e801e909f/volumes"
Oct 29 23:07:51.620477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.620440    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"]
Oct 29 23:07:51.620934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.620658    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc" podUID="11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" containerName="payload-processing" containerID="cri-o://af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6" gracePeriod=30
Oct 29 23:07:51.940569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.940547    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:07:51.975573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.975540    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-kube-api-access-5zrx5\" (UniqueName: \"kubernetes.io/projected/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-kube-api-access-5zrx5\") pod \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") "
Oct 29 23:07:51.975887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.975867    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-plugins-config-volume\") pod \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\" (UID: \"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0\") "
Oct 29 23:07:51.976217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.976195    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-plugins-config-volume" pod "11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" (UID: "11eccab3-9bc6-42f9-92c1-0a3a2f349bb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:51.976401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.976383    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:51.977753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:51.977727    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-kube-api-access-5zrx5" pod "11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" (UID: "11eccab3-9bc6-42f9-92c1-0a3a2f349bb0"). InnerVolumeSpecName "kube-api-access-5zrx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:52.098792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.092534    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrx5\" (UniqueName: \"kubernetes.io/projected/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0-kube-api-access-5zrx5\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.458364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.457547    4827 generic.go:356] "Generic (PLEG): container finished" podID="11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" containerID="af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6" exitCode=0
Oct 29 23:07:52.458364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.457840    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc" event={"ID":"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0","Type":"ContainerDied","Data":"af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6"}
Oct 29 23:07:52.458364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.457941    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc" event={"ID":"11eccab3-9bc6-42f9-92c1-0a3a2f349bb0","Type":"ContainerDied","Data":"8dfa10c895b92cbdc49006149464acd9eb6fe897bf500a7ff49b8f65c0976ea7"}
Oct 29 23:07:52.458364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.457980    4827 scope.go:122] "RemoveContainer" containerID="af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6"
Oct 29 23:07:52.458364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.458194    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"
Oct 29 23:07:52.472518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.472462    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"]
Oct 29 23:07:52.481562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.481063    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b7950-grfb7"]
Oct 29 23:07:52.499000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.498976    4827 scope.go:122] "RemoveContainer" containerID="af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6"
Oct 29 23:07:52.500190 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:07:52.500160    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6\": container with ID starting with af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6 not found: ID does not exist" containerID="af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6"
Oct 29 23:07:52.500285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.500203    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6"} err="failed to get container status \"af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6\": rpc error: code = NotFound desc = could not find container \"af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6\": container with ID starting with af73c8e4c045cb0cbd3f518ce3b4bf5cea7d90cac9ce09f0a3bf4fbfa11f1fa6 not found: ID does not exist"
Oct 29 23:07:52.510337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.510304    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e563d1c5-3de7-496f-8259-82b20c410a9f" path="/var/lib/kubelet/pods/e563d1c5-3de7-496f-8259-82b20c410a9f/volumes"
Oct 29 23:07:52.602199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.602158    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"]
Oct 29 23:07:52.608281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:52.608242    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7950-5f7db79ccd-xn7wc"]
Oct 29 23:07:54.773347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:07:54.763966    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" path="/var/lib/kubelet/pods/11eccab3-9bc6-42f9-92c1-0a3a2f349bb0/volumes"
Oct 29 23:08:00.155506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:00.155472    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"]
Oct 29 23:08:00.156000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:00.155778    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" containerID="cri-o://7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5" gracePeriod=30
Oct 29 23:08:00.166482 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:00.166449    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"]
Oct 29 23:08:00.166720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:00.166694    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy" containerID="cri-o://66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d" gracePeriod=30
Oct 29 23:08:02.194684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:02.194640    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"]
Oct 29 23:08:02.196123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:02.195747    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy" containerID="cri-o://09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f" gracePeriod=30
Oct 29 23:08:02.829787 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:02.829737    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:08:05.482524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.482497    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:08:05.535832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.535805    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:08:05.562751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.562713    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-istio-token\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-istio-token\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.562953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.562832    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-credential-socket\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.562953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.562874    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-data\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.562953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.562912    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-kube-api-access-vngps\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-kube-api-access-vngps\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.563136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.563075    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e43e343d-7c51-411b-8127-547323e065c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e343d-7c51-411b-8127-547323e065c0-istio-podinfo\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.563195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.563138    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43e343d-7c51-411b-8127-547323e065c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e343d-7c51-411b-8127-547323e065c0-istiod-ca-cert\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.563251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.563220    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-certs\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.563348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.563331    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-socket\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.563418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.563383    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-envoy\") pod \"e43e343d-7c51-411b-8127-547323e065c0\" (UID: \"e43e343d-7c51-411b-8127-547323e065c0\") "
Oct 29 23:08:05.563552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.563522    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-data" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.564151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.564127    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43e343d-7c51-411b-8127-547323e065c0-istiod-ca-cert" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:05.564270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.564216    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.564742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.564587    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-credential-socket" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.564742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.564659    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-socket" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.564742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.564698    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-certs" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.566880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.566323    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-istio-token" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:05.566880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.566575    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-kube-api-access-vngps" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "kube-api-access-vngps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:05.567228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.567204    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-envoy" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.567465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.567442    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e43e343d-7c51-411b-8127-547323e065c0-istio-podinfo" pod "e43e343d-7c51-411b-8127-547323e065c0" (UID: "e43e343d-7c51-411b-8127-547323e065c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:05.664903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.664870    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-certs\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.664949    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-token\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-token\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665004    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-socket\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665056    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-data\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665096    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-podinfo\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665152    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-kube-api-access-h5rgr\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-kube-api-access-h5rgr\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665161    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-certs" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.665319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665244    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-envoy\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665280    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-credential-socket\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665332    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istiod-ca-cert\") pod \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\" (UID: \"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b\") "
Oct 29 23:08:05.665603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665365    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-socket" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.665603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665555    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-credential-socket" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.665790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665698    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istiod-ca-cert" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:05.665908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.665885    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-data" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.666372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666350    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666371    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666382    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666394    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666402    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666409    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e343d-7c51-411b-8127-547323e065c0-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666418    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vngps\" (UniqueName: \"kubernetes.io/projected/e43e343d-7c51-411b-8127-547323e065c0-kube-api-access-vngps\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666426    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666434    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666444    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e343d-7c51-411b-8127-547323e065c0-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666453    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666465    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e343d-7c51-411b-8127-547323e065c0-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.666558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.666476    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.667802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.667748    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-token" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:05.667889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.667862    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-envoy" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:05.668445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.667905    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-kube-api-access-h5rgr" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "kube-api-access-h5rgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:05.668445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.668112    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-podinfo" pod "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" (UID: "28bcc78f-48b4-4f8a-a9ad-2c751dccb63b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:05.786946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.783893    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.786946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.783940    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5rgr\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-kube-api-access-h5rgr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.786946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.783955    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.786946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.783969    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:05.850629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.850582    4827 generic.go:356] "Generic (PLEG): container finished" podID="e43e343d-7c51-411b-8127-547323e065c0" containerID="7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5" exitCode=0
Oct 29 23:08:05.850891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.850838    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" event={"ID":"e43e343d-7c51-411b-8127-547323e065c0","Type":"ContainerDied","Data":"7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5"}
Oct 29 23:08:05.850891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.850891    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" event={"ID":"e43e343d-7c51-411b-8127-547323e065c0","Type":"ContainerDied","Data":"6456b50f2035c3840e8d9e01076dec05c4e148be1c06cee2309dbf6ba456eea2"}
Oct 29 23:08:05.851060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.850918    4827 scope.go:122] "RemoveContainer" containerID="7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5"
Oct 29 23:08:05.851134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.851074    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"
Oct 29 23:08:05.873048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.872977    4827 generic.go:356] "Generic (PLEG): container finished" podID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerID="66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d" exitCode=0
Oct 29 23:08:05.873198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.873062    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"
Oct 29 23:08:05.873198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.873064    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" event={"ID":"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b","Type":"ContainerDied","Data":"66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d"}
Oct 29 23:08:05.873198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.873185    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8" event={"ID":"28bcc78f-48b4-4f8a-a9ad-2c751dccb63b","Type":"ContainerDied","Data":"32a48313b5abf91eab479a52a8bd37b542292f0b1d025e3de985cbdfba8d3fc5"}
Oct 29 23:08:05.883358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.883343    4827 scope.go:122] "RemoveContainer" containerID="7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5"
Oct 29 23:08:05.883645 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:05.883624    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5\": container with ID starting with 7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5 not found: ID does not exist" containerID="7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5"
Oct 29 23:08:05.883699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.883654    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5"} err="failed to get container status \"7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5\": rpc error: code = NotFound desc = could not find container \"7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5\": container with ID starting with 7a30e0bf4d72e603e309fc2f5979aea25334fe5c839ab43629c6113aef908cc5 not found: ID does not exist"
Oct 29 23:08:05.883699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.883684    4827 scope.go:122] "RemoveContainer" containerID="66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d"
Oct 29 23:08:05.893380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.893362    4827 scope.go:122] "RemoveContainer" containerID="66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d"
Oct 29 23:08:05.893636 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:05.893618    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d\": container with ID starting with 66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d not found: ID does not exist" containerID="66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d"
Oct 29 23:08:05.893696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.893645    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d"} err="failed to get container status \"66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d\": rpc error: code = NotFound desc = could not find container \"66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d\": container with ID starting with 66e5150f837f81722146137cb2f798f4ef8ff6d2eb42e3089a8ffc6c858c406d not found: ID does not exist"
Oct 29 23:08:05.928049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.927974    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"]
Oct 29 23:08:05.932932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.932906    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-86md8"]
Oct 29 23:08:05.963913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.963886    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"]
Oct 29 23:08:05.969835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:05.969798    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9"]
Oct 29 23:08:06.343533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:06.343501    4827 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:08:06.343718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:06.343561    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-kfzm9" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:06.488722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:06.488632    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" path="/var/lib/kubelet/pods/28bcc78f-48b4-4f8a-a9ad-2c751dccb63b/volumes"
Oct 29 23:08:06.489368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:06.489347    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43e343d-7c51-411b-8127-547323e065c0" path="/var/lib/kubelet/pods/e43e343d-7c51-411b-8127-547323e065c0/volumes"
Oct 29 23:08:07.523063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.522534    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:08:07.610874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.610840    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-envoy\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.610891    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4230c6-f355-462c-8a8f-eb917819bd34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4230c6-f355-462c-8a8f-eb917819bd34-istiod-ca-cert\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.610936    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-kube-api-access-5wdc6\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-kube-api-access-5wdc6\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.610990    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-socket\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611025    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-data\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611056    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-podinfo\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611148    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-token\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-token\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611246    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-credential-socket\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611313    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-socket" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:07.611401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611394    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-certs\") pod \"4b4230c6-f355-462c-8a8f-eb917819bd34\" (UID: \"4b4230c6-f355-462c-8a8f-eb917819bd34\") "
Oct 29 23:08:07.611627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611416    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4230c6-f355-462c-8a8f-eb917819bd34-istiod-ca-cert" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:07.611627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611542    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-data" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:07.611759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611666    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-credential-socket" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:07.611882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.611860    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-certs" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:07.612339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.612300    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4230c6-f355-462c-8a8f-eb917819bd34-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.612454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.612345    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.612454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.612360    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.612454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.612374    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.612454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.612389    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.613471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.613446    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-envoy" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:07.614076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.614040    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-kube-api-access-5wdc6" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "kube-api-access-5wdc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:07.615328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.615308    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-token" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:07.616904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.616877    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-podinfo" pod "4b4230c6-f355-462c-8a8f-eb917819bd34" (UID: "4b4230c6-f355-462c-8a8f-eb917819bd34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:07.714044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.714015    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdc6\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-kube-api-access-5wdc6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.714044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.714043    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.714241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.714056    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.714241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.714068    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4230c6-f355-462c-8a8f-eb917819bd34-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:07.895388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.895350    4827 generic.go:356] "Generic (PLEG): container finished" podID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerID="09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f" exitCode=0
Oct 29 23:08:07.895578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.895424    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"
Oct 29 23:08:07.895578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.895480    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" event={"ID":"4b4230c6-f355-462c-8a8f-eb917819bd34","Type":"ContainerDied","Data":"09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f"}
Oct 29 23:08:07.895578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.895530    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp" event={"ID":"4b4230c6-f355-462c-8a8f-eb917819bd34","Type":"ContainerDied","Data":"d045f1395b50d68a52ed75a8b4bea27408f4be058459dce893bed1365d208cd7"}
Oct 29 23:08:07.895578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.895561    4827 scope.go:122] "RemoveContainer" containerID="09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f"
Oct 29 23:08:07.912093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.912071    4827 scope.go:122] "RemoveContainer" containerID="b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d"
Oct 29 23:08:07.924433 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.924409    4827 scope.go:122] "RemoveContainer" containerID="09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f"
Oct 29 23:08:07.924828 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:07.924804    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f\": container with ID starting with 09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f not found: ID does not exist" containerID="09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f"
Oct 29 23:08:07.924902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.924839    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f"} err="failed to get container status \"09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f\": rpc error: code = NotFound desc = could not find container \"09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f\": container with ID starting with 09f1cacf8d066f009ae25dc6d5516b0303fa319c978877bda6845dc75327b38f not found: ID does not exist"
Oct 29 23:08:07.924902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.924864    4827 scope.go:122] "RemoveContainer" containerID="b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d"
Oct 29 23:08:07.925186 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:07.925162    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d\": container with ID starting with b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d not found: ID does not exist" containerID="b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d"
Oct 29 23:08:07.925256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.925195    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d"} err="failed to get container status \"b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d\": rpc error: code = NotFound desc = could not find container \"b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d\": container with ID starting with b8d615c2bb025a6f6d5638db51a306c44ae531195a845b7328243c2a4e094b9d not found: ID does not exist"
Oct 29 23:08:07.952932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.952895    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"]
Oct 29 23:08:07.965637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:07.965598    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f7d4b-openshift-default-ff79fffb9-m4vdp"]
Oct 29 23:08:08.490280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:08.490244    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" path="/var/lib/kubelet/pods/4b4230c6-f355-462c-8a8f-eb917819bd34/volumes"
Oct 29 23:08:12.697685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:12.697652    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"]
Oct 29 23:08:12.698194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:12.698016    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm" podUID="21085ba6-e60a-4c67-a3ba-32592259b601" containerName="maas-api" containerID="cri-o://5b45bff2b0e81e2b324b96088b98f5324f787c6eda3dec3dc4ac0220869a90c4" gracePeriod=30
Oct 29 23:08:12.850731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:12.850697    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"]
Oct 29 23:08:12.850993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:12.850964    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt" podUID="9afd352a-1d5d-473d-9e13-930a469ad580" containerName="payload-processing" containerID="cri-o://3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954" gracePeriod=30
Oct 29 23:08:12.946933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:12.946068    4827 generic.go:356] "Generic (PLEG): container finished" podID="21085ba6-e60a-4c67-a3ba-32592259b601" containerID="5b45bff2b0e81e2b324b96088b98f5324f787c6eda3dec3dc4ac0220869a90c4" exitCode=0
Oct 29 23:08:12.946933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:12.946296    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm" event={"ID":"21085ba6-e60a-4c67-a3ba-32592259b601","Type":"ContainerDied","Data":"5b45bff2b0e81e2b324b96088b98f5324f787c6eda3dec3dc4ac0220869a90c4"}
Oct 29 23:08:13.042064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.041719    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:08:13.122391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.122362    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-maas-api-tls\") pod \"21085ba6-e60a-4c67-a3ba-32592259b601\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") "
Oct 29 23:08:13.122577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.122427    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21085ba6-e60a-4c67-a3ba-32592259b601-kube-api-access-7jw6s\" (UniqueName: \"kubernetes.io/projected/21085ba6-e60a-4c67-a3ba-32592259b601-kube-api-access-7jw6s\") pod \"21085ba6-e60a-4c67-a3ba-32592259b601\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") "
Oct 29 23:08:13.122665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.122632    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21085ba6-e60a-4c67-a3ba-32592259b601-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21085ba6-e60a-4c67-a3ba-32592259b601-openshift-service-ca\") pod \"21085ba6-e60a-4c67-a3ba-32592259b601\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") "
Oct 29 23:08:13.122735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.122666    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-metrics-certs\") pod \"21085ba6-e60a-4c67-a3ba-32592259b601\" (UID: \"21085ba6-e60a-4c67-a3ba-32592259b601\") "
Oct 29 23:08:13.123607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.123574    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21085ba6-e60a-4c67-a3ba-32592259b601-openshift-service-ca" pod "21085ba6-e60a-4c67-a3ba-32592259b601" (UID: "21085ba6-e60a-4c67-a3ba-32592259b601"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:13.125038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.125013    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21085ba6-e60a-4c67-a3ba-32592259b601-kube-api-access-7jw6s" pod "21085ba6-e60a-4c67-a3ba-32592259b601" (UID: "21085ba6-e60a-4c67-a3ba-32592259b601"). InnerVolumeSpecName "kube-api-access-7jw6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:13.137346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.137299    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-maas-api-tls" pod "21085ba6-e60a-4c67-a3ba-32592259b601" (UID: "21085ba6-e60a-4c67-a3ba-32592259b601"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:13.137346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.137320    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-metrics-certs" pod "21085ba6-e60a-4c67-a3ba-32592259b601" (UID: "21085ba6-e60a-4c67-a3ba-32592259b601"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:13.153583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.153557    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:08:13.232833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.231759    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21085ba6-e60a-4c67-a3ba-32592259b601-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.232833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.231813    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.232833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.231828    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21085ba6-e60a-4c67-a3ba-32592259b601-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.232833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.231841    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jw6s\" (UniqueName: \"kubernetes.io/projected/21085ba6-e60a-4c67-a3ba-32592259b601-kube-api-access-7jw6s\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.360792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.354314    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afd352a-1d5d-473d-9e13-930a469ad580-kube-api-access-xzvq4\" (UniqueName: \"kubernetes.io/projected/9afd352a-1d5d-473d-9e13-930a469ad580-kube-api-access-xzvq4\") pod \"9afd352a-1d5d-473d-9e13-930a469ad580\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") "
Oct 29 23:08:13.360792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.354420    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afd352a-1d5d-473d-9e13-930a469ad580-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd352a-1d5d-473d-9e13-930a469ad580-plugins-config-volume\") pod \"9afd352a-1d5d-473d-9e13-930a469ad580\" (UID: \"9afd352a-1d5d-473d-9e13-930a469ad580\") "
Oct 29 23:08:13.438789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.438269    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afd352a-1d5d-473d-9e13-930a469ad580-plugins-config-volume" pod "9afd352a-1d5d-473d-9e13-930a469ad580" (UID: "9afd352a-1d5d-473d-9e13-930a469ad580"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:13.473788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.460193    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd352a-1d5d-473d-9e13-930a469ad580-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.553747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.552661    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afd352a-1d5d-473d-9e13-930a469ad580-kube-api-access-xzvq4" pod "9afd352a-1d5d-473d-9e13-930a469ad580" (UID: "9afd352a-1d5d-473d-9e13-930a469ad580"). InnerVolumeSpecName "kube-api-access-xzvq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:13.561620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.561585    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzvq4\" (UniqueName: \"kubernetes.io/projected/9afd352a-1d5d-473d-9e13-930a469ad580-kube-api-access-xzvq4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.582616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.582587    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"]
Oct 29 23:08:13.586052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:13.586028    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c88955-rk8c4"]
Oct 29 23:08:14.010112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.010075    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm" event={"ID":"21085ba6-e60a-4c67-a3ba-32592259b601","Type":"ContainerDied","Data":"87b7405618a6572d69d359e6c711be49e8c2794d8eba89321721295de78d3c89"}
Oct 29 23:08:14.016265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.016237    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt" event={"ID":"9afd352a-1d5d-473d-9e13-930a469ad580","Type":"ContainerDied","Data":"3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954"}
Oct 29 23:08:14.016521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.015887    4827 generic.go:356] "Generic (PLEG): container finished" podID="9afd352a-1d5d-473d-9e13-930a469ad580" containerID="3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954" exitCode=0
Oct 29 23:08:14.016731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.016716    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt" event={"ID":"9afd352a-1d5d-473d-9e13-930a469ad580","Type":"ContainerDied","Data":"d49cd76708e82f0bd80c23e372ef31a768c703c0fc7393f525a7e166353e975d"}
Oct 29 23:08:14.017437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.011116    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"
Oct 29 23:08:14.030846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.016463    4827 scope.go:122] "RemoveContainer" containerID="5b45bff2b0e81e2b324b96088b98f5324f787c6eda3dec3dc4ac0220869a90c4"
Oct 29 23:08:14.030846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.015949    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"
Oct 29 23:08:14.075665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.075280    4827 scope.go:122] "RemoveContainer" containerID="3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954"
Oct 29 23:08:14.102843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.102505    4827 scope.go:122] "RemoveContainer" containerID="3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954"
Oct 29 23:08:14.103330 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:14.103228    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954\": container with ID starting with 3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954 not found: ID does not exist" containerID="3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954"
Oct 29 23:08:14.103330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.103270    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954"} err="failed to get container status \"3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954\": rpc error: code = NotFound desc = could not find container \"3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954\": container with ID starting with 3542d64728369cfd55a9b7ed6764d2dbb819d81cbdba1f984a37c0da09f2c954 not found: ID does not exist"
Oct 29 23:08:14.116238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.116209    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"]
Oct 29 23:08:14.145854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.144510    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c88955-7456b6c865-5xdjm"]
Oct 29 23:08:14.172792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.170640    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"]
Oct 29 23:08:14.197788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.194579    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-68846d6d48-wsvwt"]
Oct 29 23:08:14.492090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.492057    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21085ba6-e60a-4c67-a3ba-32592259b601" path="/var/lib/kubelet/pods/21085ba6-e60a-4c67-a3ba-32592259b601/volumes"
Oct 29 23:08:14.492484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.492470    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afd352a-1d5d-473d-9e13-930a469ad580" path="/var/lib/kubelet/pods/9afd352a-1d5d-473d-9e13-930a469ad580/volumes"
Oct 29 23:08:14.492887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.492874    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1581006-f0c0-49e7-bff0-bc00ef834dd0" path="/var/lib/kubelet/pods/c1581006-f0c0-49e7-bff0-bc00ef834dd0/volumes"
Oct 29 23:08:14.627477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.627449    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"]
Oct 29 23:08:14.628089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628074    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9d375c9d-14dd-4f68-920d-972e801e909f" containerName="maas-api"
Oct 29 23:08:14.628152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628090    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d375c9d-14dd-4f68-920d-972e801e909f" containerName="maas-api"
Oct 29 23:08:14.628152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628104    4827 cpu_manager.go:414] "containerMap: removing container" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy"
Oct 29 23:08:14.628152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628109    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy"
Oct 29 23:08:14.628152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628116    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy"
Oct 29 23:08:14.628152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628121    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy"
Oct 29 23:08:14.628152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628143    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy"
Oct 29 23:08:14.628152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628148    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628161    4827 cpu_manager.go:414] "containerMap: removing container" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628167    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628174    4827 cpu_manager.go:414] "containerMap: removing container" podUID="21085ba6-e60a-4c67-a3ba-32592259b601" containerName="maas-api"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628179    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21085ba6-e60a-4c67-a3ba-32592259b601" containerName="maas-api"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628188    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628192    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628203    4827 cpu_manager.go:414] "containerMap: removing container" podUID="11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" containerName="payload-processing"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628207    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" containerName="payload-processing"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628216    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9afd352a-1d5d-473d-9e13-930a469ad580" containerName="payload-processing"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628221    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afd352a-1d5d-473d-9e13-930a469ad580" containerName="payload-processing"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628230    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ce5363d4-9c46-4624-87a5-d1dae17e1b19" containerName="revoke-keys"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628234    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5363d4-9c46-4624-87a5-d1dae17e1b19" containerName="revoke-keys"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628310    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d375c9d-14dd-4f68-920d-972e801e909f" containerName="maas-api"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628318    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bcc78f-48b4-4f8a-a9ad-2c751dccb63b" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628324    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43e343d-7c51-411b-8127-547323e065c0" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628332    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628338    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd3d0b2-dc85-4627-a891-0c5d33639748" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628344    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="21085ba6-e60a-4c67-a3ba-32592259b601" containerName="maas-api"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628350    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4230c6-f355-462c-8a8f-eb917819bd34" containerName="istio-proxy"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628356    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="11eccab3-9bc6-42f9-92c1-0a3a2f349bb0" containerName="payload-processing"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628363    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afd352a-1d5d-473d-9e13-930a469ad580" containerName="payload-processing"
Oct 29 23:08:14.628371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.628367    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5363d4-9c46-4624-87a5-d1dae17e1b19" containerName="revoke-keys"
Oct 29 23:08:14.633209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.633191    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.665388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.665360    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"]
Oct 29 23:08:14.672238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.672211    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:14.672357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.672264    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:08:14.728991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.728959    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ms9n\" (UniqueName: \"kubernetes.io/projected/fae11e6d-c37a-4b0e-9de7-a746713a8840-kube-api-access-8ms9n\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.729186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.729026    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.729186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.729149    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.830427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.830348    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ms9n\" (UniqueName: \"kubernetes.io/projected/fae11e6d-c37a-4b0e-9de7-a746713a8840-kube-api-access-8ms9n\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.830427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.830415    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.830623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.830478    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.831128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.831106    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.832924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.832898    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.853538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.853506    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ms9n\" (UniqueName: \"kubernetes.io/projected/fae11e6d-c37a-4b0e-9de7-a746713a8840-kube-api-access-8ms9n\") pod \"maas-api-revoke-sub-415742ef-rzl8x\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:14.968291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:14.968258    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:15.392839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:15.372890    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"]
Oct 29 23:08:16.036421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:16.036321    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x" event={"ID":"fae11e6d-c37a-4b0e-9de7-a746713a8840","Type":"ContainerStarted","Data":"f9e9b4d089ae048a28e8ed09c8782fb2f554e4380114537dbe868f16bbb3ea60"}
Oct 29 23:08:16.036421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:16.036379    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x" event={"ID":"fae11e6d-c37a-4b0e-9de7-a746713a8840","Type":"ContainerStarted","Data":"3e83fbc4bc3bde678f9de19a6f6ff712cca9c1dd28a76aff36c05ca7d6bfaf91"}
Oct 29 23:08:16.077014 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:16.076947    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x" podStartSLOduration=2.076928767 podStartE2EDuration="2.076928767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:16.075239779 +0000 UTC m=+1531.698150799" watchObservedRunningTime="2026-10-29 23:08:16.076928767 +0000 UTC m=+1531.699839788"
Oct 29 23:08:17.045875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:17.045842    4827 generic.go:356] "Generic (PLEG): container finished" podID="fae11e6d-c37a-4b0e-9de7-a746713a8840" containerID="f9e9b4d089ae048a28e8ed09c8782fb2f554e4380114537dbe868f16bbb3ea60" exitCode=0
Oct 29 23:08:17.046289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:17.045921    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x" event={"ID":"fae11e6d-c37a-4b0e-9de7-a746713a8840","Type":"ContainerDied","Data":"f9e9b4d089ae048a28e8ed09c8782fb2f554e4380114537dbe868f16bbb3ea60"}
Oct 29 23:08:18.227812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.227787    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:18.311860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.311779    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae11e6d-c37a-4b0e-9de7-a746713a8840-kube-api-access-8ms9n\" (UniqueName: \"kubernetes.io/projected/fae11e6d-c37a-4b0e-9de7-a746713a8840-kube-api-access-8ms9n\") pod \"fae11e6d-c37a-4b0e-9de7-a746713a8840\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") "
Oct 29 23:08:18.312035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.311869    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-xks-ca\") pod \"fae11e6d-c37a-4b0e-9de7-a746713a8840\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") "
Oct 29 23:08:18.312035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.311968    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-openshift-ca\") pod \"fae11e6d-c37a-4b0e-9de7-a746713a8840\" (UID: \"fae11e6d-c37a-4b0e-9de7-a746713a8840\") "
Oct 29 23:08:18.312363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.312332    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-openshift-ca" pod "fae11e6d-c37a-4b0e-9de7-a746713a8840" (UID: "fae11e6d-c37a-4b0e-9de7-a746713a8840"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:18.312685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.312667    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:18.314212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.314183    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-xks-ca" pod "fae11e6d-c37a-4b0e-9de7-a746713a8840" (UID: "fae11e6d-c37a-4b0e-9de7-a746713a8840"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:18.314522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.314494    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae11e6d-c37a-4b0e-9de7-a746713a8840-kube-api-access-8ms9n" pod "fae11e6d-c37a-4b0e-9de7-a746713a8840" (UID: "fae11e6d-c37a-4b0e-9de7-a746713a8840"). InnerVolumeSpecName "kube-api-access-8ms9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:18.414322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.414278    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ms9n\" (UniqueName: \"kubernetes.io/projected/fae11e6d-c37a-4b0e-9de7-a746713a8840-kube-api-access-8ms9n\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:18.414322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:18.414314    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae11e6d-c37a-4b0e-9de7-a746713a8840-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:19.142813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:19.132041    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x" event={"ID":"fae11e6d-c37a-4b0e-9de7-a746713a8840","Type":"ContainerDied","Data":"3e83fbc4bc3bde678f9de19a6f6ff712cca9c1dd28a76aff36c05ca7d6bfaf91"}
Oct 29 23:08:19.142813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:19.132107    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e83fbc4bc3bde678f9de19a6f6ff712cca9c1dd28a76aff36c05ca7d6bfaf91"
Oct 29 23:08:19.142813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:19.132176    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"
Oct 29 23:08:20.918739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:20.918702    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"]
Oct 29 23:08:20.919608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:20.919569    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy" containerID="cri-o://4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c" gracePeriod=30
Oct 29 23:08:25.490443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.490402    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"]
Oct 29 23:08:25.491009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.490630    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx" podUID="59a79136-7646-4227-bac1-328e00707460" containerName="maas-api" containerID="cri-o://9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228" gracePeriod=30
Oct 29 23:08:25.775880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.775855    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:08:25.871325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.871294    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a79136-7646-4227-bac1-328e00707460-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a79136-7646-4227-bac1-328e00707460-openshift-service-ca\") pod \"59a79136-7646-4227-bac1-328e00707460\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") "
Oct 29 23:08:25.871603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.871576    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a79136-7646-4227-bac1-328e00707460-kube-api-access-dkntx\" (UniqueName: \"kubernetes.io/projected/59a79136-7646-4227-bac1-328e00707460-kube-api-access-dkntx\") pod \"59a79136-7646-4227-bac1-328e00707460\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") "
Oct 29 23:08:25.871728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.871598    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a79136-7646-4227-bac1-328e00707460-openshift-service-ca" pod "59a79136-7646-4227-bac1-328e00707460" (UID: "59a79136-7646-4227-bac1-328e00707460"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:25.871728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.871650    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-maas-api-tls\") pod \"59a79136-7646-4227-bac1-328e00707460\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") "
Oct 29 23:08:25.872016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.871854    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-metrics-certs\") pod \"59a79136-7646-4227-bac1-328e00707460\" (UID: \"59a79136-7646-4227-bac1-328e00707460\") "
Oct 29 23:08:25.873010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.872978    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a79136-7646-4227-bac1-328e00707460-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:25.874954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.874914    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a79136-7646-4227-bac1-328e00707460-kube-api-access-dkntx" pod "59a79136-7646-4227-bac1-328e00707460" (UID: "59a79136-7646-4227-bac1-328e00707460"). InnerVolumeSpecName "kube-api-access-dkntx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:25.875355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.875310    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-metrics-certs" pod "59a79136-7646-4227-bac1-328e00707460" (UID: "59a79136-7646-4227-bac1-328e00707460"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:25.876490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.876437    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-maas-api-tls" pod "59a79136-7646-4227-bac1-328e00707460" (UID: "59a79136-7646-4227-bac1-328e00707460"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:25.974985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.974957    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkntx\" (UniqueName: \"kubernetes.io/projected/59a79136-7646-4227-bac1-328e00707460-kube-api-access-dkntx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:25.974985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.974984    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:25.975167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:25.974997    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a79136-7646-4227-bac1-328e00707460-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.230090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.230055    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:08:26.269562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.269529    4827 generic.go:356] "Generic (PLEG): container finished" podID="59a79136-7646-4227-bac1-328e00707460" containerID="9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228" exitCode=0
Oct 29 23:08:26.269795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.269743    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx" event={"ID":"59a79136-7646-4227-bac1-328e00707460","Type":"ContainerDied","Data":"9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228"}
Oct 29 23:08:26.269892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.269815    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx" event={"ID":"59a79136-7646-4227-bac1-328e00707460","Type":"ContainerDied","Data":"53d64574230e61a89a44130330dead6fdfec5b2b5f0caaf96948b464536cc5a7"}
Oct 29 23:08:26.269892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.269848    4827 scope.go:122] "RemoveContainer" containerID="9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228"
Oct 29 23:08:26.270026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.270011    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290074    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-token\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-token\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290217    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-socket\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290283    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-envoy\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290331    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290377    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290408    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-data\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290470    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-credential-socket\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290529    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-certs\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.320877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.290590    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-kube-api-access-kz5cq\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-kube-api-access-kz5cq\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.688723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.687242    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:26.689264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.688094    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:26.694472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.694423    4827 generic.go:356] "Generic (PLEG): container finished" podID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerID="4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c" exitCode=0
Oct 29 23:08:26.694679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.694660    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"
Oct 29 23:08:26.707788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.703204    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.712762 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.708005    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\") pod \"c0ba4855-3fc6-48d4-81da-448a9ff517af\" (UID: \"c0ba4855-3fc6-48d4-81da-448a9ff517af\") "
Oct 29 23:08:26.713680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.713143    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-kube-api-access-kz5cq" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "kube-api-access-kz5cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:26.727879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.718348    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-token" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:26.727879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.727862    4827 scope.go:122] "RemoveContainer" containerID="9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228"
Oct 29 23:08:26.728197 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:26.728168    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228\": container with ID starting with 9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228 not found: ID does not exist" containerID="9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228"
Oct 29 23:08:26.728258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.728199    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228"} err="failed to get container status \"9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228\": rpc error: code = NotFound desc = could not find container \"9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228\": container with ID starting with 9b27f7a6343abe7dc381fa9a3d9741d7f8c59133abb4f935e55443692b8e7228 not found: ID does not exist"
Oct 29 23:08:26.729200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.729178    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-credential-socket" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:26.729427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.729412    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-data" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:26.729559 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:26.729550    4827 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c0ba4855-3fc6-48d4-81da-448a9ff517af/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 29 23:08:26.729642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.729629    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:26.735616 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:26.729842    4827 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c0ba4855-3fc6-48d4-81da-448a9ff517af/volumes/kubernetes.io~configmap/istiod-ca-cert
Oct 29 23:08:26.735616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.729859    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:26.735616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.730101    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-certs" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:26.735616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.730281    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-socket" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:26.737728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.736829    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-envoy" pod "c0ba4855-3fc6-48d4-81da-448a9ff517af" (UID: "c0ba4855-3fc6-48d4-81da-448a9ff517af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:26.737728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.737057    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" event={"ID":"c0ba4855-3fc6-48d4-81da-448a9ff517af","Type":"ContainerDied","Data":"4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c"}
Oct 29 23:08:26.737728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.737112    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2" event={"ID":"c0ba4855-3fc6-48d4-81da-448a9ff517af","Type":"ContainerDied","Data":"1dc71b329c18a15677456df3ec757fc665e234e6817c1be059cf9ab1508fc6da"}
Oct 29 23:08:26.737728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.737144    4827 scope.go:122] "RemoveContainer" containerID="4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c"
Oct 29 23:08:26.770818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.770756    4827 scope.go:122] "RemoveContainer" containerID="7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da"
Oct 29 23:08:26.800056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.799986    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"]
Oct 29 23:08:26.806143 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.806121    4827 scope.go:122] "RemoveContainer" containerID="4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c"
Oct 29 23:08:26.806674 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:26.806650    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c\": container with ID starting with 4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c not found: ID does not exist" containerID="4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c"
Oct 29 23:08:26.806736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.806685    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c"} err="failed to get container status \"4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c\": rpc error: code = NotFound desc = could not find container \"4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c\": container with ID starting with 4d737a3a748acbce92afc9670cbc5dede1c94a15534dc23a894d6420b937635c not found: ID does not exist"
Oct 29 23:08:26.806736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.806709    4827 scope.go:122] "RemoveContainer" containerID="7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da"
Oct 29 23:08:26.808348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.808328    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-047892-56fcb69449-nv5fx"]
Oct 29 23:08:26.808561 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:26.808536    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da\": container with ID starting with 7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da not found: ID does not exist" containerID="7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da"
Oct 29 23:08:26.808631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.808569    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da"} err="failed to get container status \"7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da\": rpc error: code = NotFound desc = could not find container \"7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da\": container with ID starting with 7ad3345a18c4c6ece83bf5add0fc43dae5095538eb159f8af4cab9f1872d22da not found: ID does not exist"
Oct 29 23:08:26.811146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811122    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811161    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811175    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811189    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811202    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ba4855-3fc6-48d4-81da-448a9ff517af-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811215    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811227    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811240    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ba4855-3fc6-48d4-81da-448a9ff517af-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.811262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:26.811255    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz5cq\" (UniqueName: \"kubernetes.io/projected/c0ba4855-3fc6-48d4-81da-448a9ff517af-kube-api-access-kz5cq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:27.041780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:27.041734    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"]
Oct 29 23:08:27.047701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:27.047674    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c88955-openshift-default-5d9bd7686f-8twg2"]
Oct 29 23:08:28.488972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:28.488931    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a79136-7646-4227-bac1-328e00707460" path="/var/lib/kubelet/pods/59a79136-7646-4227-bac1-328e00707460/volumes"
Oct 29 23:08:28.489652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:28.489630    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" path="/var/lib/kubelet/pods/c0ba4855-3fc6-48d4-81da-448a9ff517af/volumes"
Oct 29 23:08:29.693639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.693608    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"]
Oct 29 23:08:29.694212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694198    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fae11e6d-c37a-4b0e-9de7-a746713a8840" containerName="revoke-keys"
Oct 29 23:08:29.694263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694213    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae11e6d-c37a-4b0e-9de7-a746713a8840" containerName="revoke-keys"
Oct 29 23:08:29.694263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694241    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy"
Oct 29 23:08:29.694263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694247    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy"
Oct 29 23:08:29.694263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694254    4827 cpu_manager.go:414] "containerMap: removing container" podUID="59a79136-7646-4227-bac1-328e00707460" containerName="maas-api"
Oct 29 23:08:29.694263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694259    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a79136-7646-4227-bac1-328e00707460" containerName="maas-api"
Oct 29 23:08:29.694430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694269    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy"
Oct 29 23:08:29.694430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694273    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy"
Oct 29 23:08:29.694430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694359    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a79136-7646-4227-bac1-328e00707460" containerName="maas-api"
Oct 29 23:08:29.694430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694367    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy"
Oct 29 23:08:29.694430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.694385    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae11e6d-c37a-4b0e-9de7-a746713a8840" containerName="revoke-keys"
Oct 29 23:08:29.699505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.699485    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.718117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.718093    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"]
Oct 29 23:08:29.725646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.725625    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:29.725755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.725724    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:08:29.752503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.752466    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.752643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.752556    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzwb\" (UniqueName: \"kubernetes.io/projected/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-kube-api-access-lnzwb\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.752643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.752614    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.854229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.854206    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.854439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.854276    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzwb\" (UniqueName: \"kubernetes.io/projected/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-kube-api-access-lnzwb\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.854439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.854350    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.855228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.855201    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.857456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.857434    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:29.877455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:29.877433    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzwb\" (UniqueName: \"kubernetes.io/projected/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-kube-api-access-lnzwb\") pod \"maas-api-revoke-sub-e1c7cef5-5scvd\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:30.013844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:30.012964    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:30.544539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:30.544511    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"]
Oct 29 23:08:30.545937 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:30.545911    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc5d5f2_53e9_4ef2_8a04_494293167c6b.slice/crio-4af449013c3ad4de26c56ea490ddb9a450e797e8edf0373b8d7e4b7a4dcf6796 WatchSource:0}: Error finding container 4af449013c3ad4de26c56ea490ddb9a450e797e8edf0373b8d7e4b7a4dcf6796: Status 404 returned error can't find the container with id 4af449013c3ad4de26c56ea490ddb9a450e797e8edf0373b8d7e4b7a4dcf6796
Oct 29 23:08:30.724519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:30.724487    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd" event={"ID":"8dc5d5f2-53e9-4ef2-8a04-494293167c6b","Type":"ContainerStarted","Data":"4af449013c3ad4de26c56ea490ddb9a450e797e8edf0373b8d7e4b7a4dcf6796"}
Oct 29 23:08:31.198063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.197989    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"]
Oct 29 23:08:31.198691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.198677    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ba4855-3fc6-48d4-81da-448a9ff517af" containerName="istio-proxy"
Oct 29 23:08:31.202170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.202150    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.223194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.223167    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"]
Oct 29 23:08:31.289073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.289041    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be891873-b788-49ad-b545-a3fa4842851b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.289253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.289086    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be891873-b788-49ad-b545-a3fa4842851b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.289356 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.289332    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/be891873-b788-49ad-b545-a3fa4842851b-kube-api-access-kh9jc\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.390367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.390335    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/be891873-b788-49ad-b545-a3fa4842851b-kube-api-access-kh9jc\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.390557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.390487    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be891873-b788-49ad-b545-a3fa4842851b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.390557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.390525    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be891873-b788-49ad-b545-a3fa4842851b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.391248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.391224    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be891873-b788-49ad-b545-a3fa4842851b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.392966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.392948    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be891873-b788-49ad-b545-a3fa4842851b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.461574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.441566    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/be891873-b788-49ad-b545-a3fa4842851b-kube-api-access-kh9jc\") pod \"maas-api-revoke-sub-7f76da0b-n6tls\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.518562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.518303    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:31.741235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.741121    4827 generic.go:356] "Generic (PLEG): container finished" podID="8dc5d5f2-53e9-4ef2-8a04-494293167c6b" containerID="f94e5338d05d51e5943585592740d427cd946492b3a3ea2b0c6cc679334d0baa" exitCode=0
Oct 29 23:08:31.741235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.741191    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd" event={"ID":"8dc5d5f2-53e9-4ef2-8a04-494293167c6b","Type":"ContainerDied","Data":"f94e5338d05d51e5943585592740d427cd946492b3a3ea2b0c6cc679334d0baa"}
Oct 29 23:08:31.976644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:31.976164    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"]
Oct 29 23:08:32.750700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:32.750664    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls" event={"ID":"be891873-b788-49ad-b545-a3fa4842851b","Type":"ContainerStarted","Data":"65c453f407d9eb76c2a01faf5ea947387abea7f6aeb7a42cc0ae6b1e231f6b30"}
Oct 29 23:08:32.751190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:32.750715    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls" event={"ID":"be891873-b788-49ad-b545-a3fa4842851b","Type":"ContainerStarted","Data":"308b9ed75794263ce69f8fab4bdc9a766feb4cefa88772044e11927960d64d64"}
Oct 29 23:08:32.805556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:32.805503    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls" podStartSLOduration=1.805489557 podStartE2EDuration="1.805489557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:32.791137007 +0000 UTC m=+1548.414048037" watchObservedRunningTime="2026-10-29 23:08:32.805489557 +0000 UTC m=+1548.428400578"
Oct 29 23:08:32.833091 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:32.833054    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:08:32.939945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:32.939920    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:33.048839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.048739    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-openshift-ca\") pod \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") "
Oct 29 23:08:33.048998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.048882    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-kube-api-access-lnzwb\" (UniqueName: \"kubernetes.io/projected/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-kube-api-access-lnzwb\") pod \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") "
Oct 29 23:08:33.049060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.049045    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-xks-ca\") pod \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\" (UID: \"8dc5d5f2-53e9-4ef2-8a04-494293167c6b\") "
Oct 29 23:08:33.049189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.049168    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-openshift-ca" pod "8dc5d5f2-53e9-4ef2-8a04-494293167c6b" (UID: "8dc5d5f2-53e9-4ef2-8a04-494293167c6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:33.049559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.049535    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:33.050966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.050941    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-kube-api-access-lnzwb" pod "8dc5d5f2-53e9-4ef2-8a04-494293167c6b" (UID: "8dc5d5f2-53e9-4ef2-8a04-494293167c6b"). InnerVolumeSpecName "kube-api-access-lnzwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:33.051076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.051056    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-xks-ca" pod "8dc5d5f2-53e9-4ef2-8a04-494293167c6b" (UID: "8dc5d5f2-53e9-4ef2-8a04-494293167c6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:33.150542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.150509    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:33.150542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.150535    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzwb\" (UniqueName: \"kubernetes.io/projected/8dc5d5f2-53e9-4ef2-8a04-494293167c6b-kube-api-access-lnzwb\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:33.759108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.759074    4827 generic.go:356] "Generic (PLEG): container finished" podID="be891873-b788-49ad-b545-a3fa4842851b" containerID="65c453f407d9eb76c2a01faf5ea947387abea7f6aeb7a42cc0ae6b1e231f6b30" exitCode=0
Oct 29 23:08:33.759560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.759157    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls" event={"ID":"be891873-b788-49ad-b545-a3fa4842851b","Type":"ContainerDied","Data":"65c453f407d9eb76c2a01faf5ea947387abea7f6aeb7a42cc0ae6b1e231f6b30"}
Oct 29 23:08:33.760588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.760567    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"
Oct 29 23:08:33.760714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.760585    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd" event={"ID":"8dc5d5f2-53e9-4ef2-8a04-494293167c6b","Type":"ContainerDied","Data":"4af449013c3ad4de26c56ea490ddb9a450e797e8edf0373b8d7e4b7a4dcf6796"}
Oct 29 23:08:33.760714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:33.760626    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af449013c3ad4de26c56ea490ddb9a450e797e8edf0373b8d7e4b7a4dcf6796"
Oct 29 23:08:34.551613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.551580    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"]
Oct 29 23:08:34.552185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.552129    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" containerID="cri-o://c28f89e53885e6d18f93c9d00a0dc5b0b0dee057a9d307cbf719fad352c0ec30" gracePeriod=30
Oct 29 23:08:34.628831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.624889    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"]
Oct 29 23:08:34.628831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.625153    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" containerID="cri-o://3ebe0e778b340f6b12dd3c7f150e18b3e10238c20b21147fdd7a1596c50e36ea" gracePeriod=30
Oct 29 23:08:34.628831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.628221    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"]
Oct 29 23:08:34.628831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.628427    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" containerID="cri-o://0d350dc35f7d683da0453be8fbf55a134d26d16875bff1498d6f68710068473f" gracePeriod=30
Oct 29 23:08:34.951224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.951193    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:08:34.951726 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:34.951252    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:08:35.037284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.036895    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:08:35.037284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.036972    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:08:35.094242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.094171    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:35.246085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.246055    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be891873-b788-49ad-b545-a3fa4842851b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be891873-b788-49ad-b545-a3fa4842851b-maas-api-xks-ca\") pod \"be891873-b788-49ad-b545-a3fa4842851b\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") "
Oct 29 23:08:35.246275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.246179    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be891873-b788-49ad-b545-a3fa4842851b-kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/be891873-b788-49ad-b545-a3fa4842851b-kube-api-access-kh9jc\") pod \"be891873-b788-49ad-b545-a3fa4842851b\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") "
Oct 29 23:08:35.246275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.246210    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be891873-b788-49ad-b545-a3fa4842851b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be891873-b788-49ad-b545-a3fa4842851b-maas-api-openshift-ca\") pod \"be891873-b788-49ad-b545-a3fa4842851b\" (UID: \"be891873-b788-49ad-b545-a3fa4842851b\") "
Oct 29 23:08:35.246629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.246600    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be891873-b788-49ad-b545-a3fa4842851b-maas-api-openshift-ca" pod "be891873-b788-49ad-b545-a3fa4842851b" (UID: "be891873-b788-49ad-b545-a3fa4842851b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:35.248221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.248196    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be891873-b788-49ad-b545-a3fa4842851b-maas-api-xks-ca" pod "be891873-b788-49ad-b545-a3fa4842851b" (UID: "be891873-b788-49ad-b545-a3fa4842851b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:35.248507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.248471    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be891873-b788-49ad-b545-a3fa4842851b-kube-api-access-kh9jc" pod "be891873-b788-49ad-b545-a3fa4842851b" (UID: "be891873-b788-49ad-b545-a3fa4842851b"). InnerVolumeSpecName "kube-api-access-kh9jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:35.348228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.348153    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/be891873-b788-49ad-b545-a3fa4842851b-kube-api-access-kh9jc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:35.348228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.348179    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be891873-b788-49ad-b545-a3fa4842851b-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:35.348228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.348190    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be891873-b788-49ad-b545-a3fa4842851b-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:35.834469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.834436    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"
Oct 29 23:08:35.834469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.834447    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls" event={"ID":"be891873-b788-49ad-b545-a3fa4842851b","Type":"ContainerDied","Data":"308b9ed75794263ce69f8fab4bdc9a766feb4cefa88772044e11927960d64d64"}
Oct 29 23:08:35.834752 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:35.834501    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="308b9ed75794263ce69f8fab4bdc9a766feb4cefa88772044e11927960d64d64"
Oct 29 23:08:37.906837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:37.905964    4827 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:08:37.906837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:37.906025    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:08:38.578791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.575605    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"]
Oct 29 23:08:38.578791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.576532    4827 cpu_manager.go:414] "containerMap: removing container" podUID="be891873-b788-49ad-b545-a3fa4842851b" containerName="revoke-keys"
Oct 29 23:08:38.578791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.576557    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be891873-b788-49ad-b545-a3fa4842851b" containerName="revoke-keys"
Oct 29 23:08:38.578791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.576608    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8dc5d5f2-53e9-4ef2-8a04-494293167c6b" containerName="revoke-keys"
Oct 29 23:08:38.578791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.576716    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc5d5f2-53e9-4ef2-8a04-494293167c6b" containerName="revoke-keys"
Oct 29 23:08:38.578791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.576951    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="be891873-b788-49ad-b545-a3fa4842851b" containerName="revoke-keys"
Oct 29 23:08:38.578791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.576975    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc5d5f2-53e9-4ef2-8a04-494293167c6b" containerName="revoke-keys"
Oct 29 23:08:38.594979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.593708    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"]
Oct 29 23:08:38.596917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.596866    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.598923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.598904    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"]
Oct 29 23:08:38.599039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.598941    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"]
Oct 29 23:08:38.599039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.599015    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.612661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.612632    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-063b46-openshift-default-dockercfg-hj7tc\""
Oct 29 23:08:38.676047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676012    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjtj\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-kube-api-access-txjtj\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.676260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676069    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.676352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676285    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jb2\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-kube-api-access-85jb2\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.676352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676339    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.676449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676415    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.676494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676457    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.676494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676487    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.676594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676537    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.676673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676653    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.676746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676695    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.676833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676752    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.676932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676915    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.676994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676949    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.677043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.676992    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.677085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.677058    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.677129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.677112    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6713f2b-a0be-46ed-a304-513aef0121dc-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.677219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.677205    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.677284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.677267    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.778391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778355    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.778600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778401    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.778600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778542    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.778600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778592    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.778821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778622    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.778821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778667    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.778821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778708    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.778821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778727    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.778821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778755    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.778821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778795    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778855    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6713f2b-a0be-46ed-a304-513aef0121dc-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778912    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.778981    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779063    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779076    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjtj\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-kube-api-access-txjtj\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779112    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779248    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779304    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779400    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779421    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779063    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.779509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779122    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779526    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jb2\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-kube-api-access-85jb2\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779573    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779616    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779648    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.779906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.779684    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6713f2b-a0be-46ed-a304-513aef0121dc-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.780284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.780258    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.781243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.781221    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.781346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.781253    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.782322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.782301    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.782410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.782393    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.801945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.801918    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.802272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.802247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.802572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.802545    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jb2\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-kube-api-access-85jb2\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.802879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.802861    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjtj\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-kube-api-access-txjtj\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.815958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.815937    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"]
Oct 29 23:08:38.822367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.822348    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:38.836094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.836073    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"]
Oct 29 23:08:38.913061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.913007    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:38.913481 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.913179    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:08:38.924260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.924231    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:38.929639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.929616    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:38.985534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.983367    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:38.985534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.983453    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xpdq\" (UniqueName: \"kubernetes.io/projected/b5fda436-26f7-4579-8456-e804c9f0fb64-kube-api-access-9xpdq\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:38.985534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:38.983550    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.097347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.097271    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.097503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.097360    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xpdq\" (UniqueName: \"kubernetes.io/projected/b5fda436-26f7-4579-8456-e804c9f0fb64-kube-api-access-9xpdq\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.097503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.097461    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.098227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.098199    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.100076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.100048    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.121036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.121012    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xpdq\" (UniqueName: \"kubernetes.io/projected/b5fda436-26f7-4579-8456-e804c9f0fb64-kube-api-access-9xpdq\") pod \"maas-api-revoke-keys-e2e-praxis-b-182195-46njr\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.136873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.136848    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:39.255327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.255293    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"]
Oct 29 23:08:39.259944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.259916    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"]
Oct 29 23:08:39.260037 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:39.259923    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983a74fa_e694_47c2_8b53_dc3f77d1aedd.slice/crio-ca503c57f0cd0e78cb5c4737ced2907bfd4ebd1cabd516fea1373dc7d5d8a831 WatchSource:0}: Error finding container ca503c57f0cd0e78cb5c4737ced2907bfd4ebd1cabd516fea1373dc7d5d8a831: Status 404 returned error can't find the container with id ca503c57f0cd0e78cb5c4737ced2907bfd4ebd1cabd516fea1373dc7d5d8a831
Oct 29 23:08:39.261930 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:39.261906    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6713f2b_a0be_46ed_a304_513aef0121dc.slice/crio-9c4a377e5fdc1298678bbad4c31ecbc0b0ba10c6cd5f3fca69e63ec8d8226bc7 WatchSource:0}: Error finding container 9c4a377e5fdc1298678bbad4c31ecbc0b0ba10c6cd5f3fca69e63ec8d8226bc7: Status 404 returned error can't find the container with id 9c4a377e5fdc1298678bbad4c31ecbc0b0ba10c6cd5f3fca69e63ec8d8226bc7
Oct 29 23:08:39.263192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.263156    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:08:39.263303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.263231    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:08:39.263303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.263276    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:08:39.264733 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.264700    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:08:39.264860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.264816    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:08:39.265480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.265441    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:08:39.462596 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:39.456449    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fda436_26f7_4579_8456_e804c9f0fb64.slice/crio-1e2e4521119bc15f502829f3595aa7273252b76aca9224abfb97f0f434ab0da9 WatchSource:0}: Error finding container 1e2e4521119bc15f502829f3595aa7273252b76aca9224abfb97f0f434ab0da9: Status 404 returned error can't find the container with id 1e2e4521119bc15f502829f3595aa7273252b76aca9224abfb97f0f434ab0da9
Oct 29 23:08:39.462596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:39.457903    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"]
Oct 29 23:08:40.131116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.129404    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" event={"ID":"d6713f2b-a0be-46ed-a304-513aef0121dc","Type":"ContainerStarted","Data":"a98304eff68928584be71b579c42c7281c56901b6eab77c32665022504f4b281"}
Oct 29 23:08:40.131116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.129470    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" event={"ID":"d6713f2b-a0be-46ed-a304-513aef0121dc","Type":"ContainerStarted","Data":"9c4a377e5fdc1298678bbad4c31ecbc0b0ba10c6cd5f3fca69e63ec8d8226bc7"}
Oct 29 23:08:40.148726 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.148602    4827 generic.go:356] "Generic (PLEG): container finished" podID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerID="c28f89e53885e6d18f93c9d00a0dc5b0b0dee057a9d307cbf719fad352c0ec30" exitCode=0
Oct 29 23:08:40.148907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.148783    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" event={"ID":"6b649c51-546f-4ad2-b36d-ddd215dcb8f7","Type":"ContainerDied","Data":"c28f89e53885e6d18f93c9d00a0dc5b0b0dee057a9d307cbf719fad352c0ec30"}
Oct 29 23:08:40.152817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.152433    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr" event={"ID":"b5fda436-26f7-4579-8456-e804c9f0fb64","Type":"ContainerStarted","Data":"1e2e4521119bc15f502829f3595aa7273252b76aca9224abfb97f0f434ab0da9"}
Oct 29 23:08:40.155556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.155403    4827 generic.go:356] "Generic (PLEG): container finished" podID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerID="0d350dc35f7d683da0453be8fbf55a134d26d16875bff1498d6f68710068473f" exitCode=0
Oct 29 23:08:40.155556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.155484    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" event={"ID":"c2fdb194-be85-4d72-9df4-fd9ec332ef3f","Type":"ContainerDied","Data":"0d350dc35f7d683da0453be8fbf55a134d26d16875bff1498d6f68710068473f"}
Oct 29 23:08:40.157972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.157468    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" event={"ID":"983a74fa-e694-47c2-8b53-dc3f77d1aedd","Type":"ContainerStarted","Data":"3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2"}
Oct 29 23:08:40.157972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.157522    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" event={"ID":"983a74fa-e694-47c2-8b53-dc3f77d1aedd","Type":"ContainerStarted","Data":"ca503c57f0cd0e78cb5c4737ced2907bfd4ebd1cabd516fea1373dc7d5d8a831"}
Oct 29 23:08:40.194811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.187358    4827 generic.go:356] "Generic (PLEG): container finished" podID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerID="3ebe0e778b340f6b12dd3c7f150e18b3e10238c20b21147fdd7a1596c50e36ea" exitCode=0
Oct 29 23:08:40.194811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.187467    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" event={"ID":"739f2a27-1668-47cf-9b3e-5beab9d7cf1d","Type":"ContainerDied","Data":"3ebe0e778b340f6b12dd3c7f150e18b3e10238c20b21147fdd7a1596c50e36ea"}
Oct 29 23:08:40.194811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.187524    4827 scope.go:122] "RemoveContainer" containerID="e2df71419852a05589c3650983941fc519f123096b9afb8afe52118cb159a18a"
Oct 29 23:08:40.235402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.235199    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podStartSLOduration=2.235180823 podStartE2EDuration="2.235180823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:40.225946405 +0000 UTC m=+1555.848857427" watchObservedRunningTime="2026-10-29 23:08:40.235180823 +0000 UTC m=+1555.858091844"
Oct 29 23:08:40.333329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.332813    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr" podStartSLOduration=2.332794276 podStartE2EDuration="2.332794276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:40.329258054 +0000 UTC m=+1555.952169076" watchObservedRunningTime="2026-10-29 23:08:40.332794276 +0000 UTC m=+1555.955705292"
Oct 29 23:08:40.375022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.374990    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:08:40.447815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.431733    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podStartSLOduration=2.431708125 podStartE2EDuration="2.431708125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:40.400102926 +0000 UTC m=+1556.023013947" watchObservedRunningTime="2026-10-29 23:08:40.431708125 +0000 UTC m=+1556.054619146"
Oct 29 23:08:40.538978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.538940    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:08:40.539243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539220    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-data\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539260    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-envoy\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539288    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-kube-api-access-h5vk5\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-kube-api-access-h5vk5\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539336    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-credential-socket\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539441    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-token\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-token\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539545 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539527    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-socket\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539550    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-certs\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539616    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istiod-ca-cert\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.539734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.539686    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-podinfo\") pod \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\" (UID: \"6b649c51-546f-4ad2-b36d-ddd215dcb8f7\") "
Oct 29 23:08:40.541038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.540870    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-certs" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.541204 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.541181    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-socket" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.541507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.541373    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-credential-socket" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.541786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.541683    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-data" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.541786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.541738    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istiod-ca-cert" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:40.544633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.544610    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-token" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:40.544909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.544892    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-podinfo" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:40.545322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.545240    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:08:40.545553 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.545530    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-envoy" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.546361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.546333    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-kube-api-access-h5vk5" pod "6b649c51-546f-4ad2-b36d-ddd215dcb8f7" (UID: "6b649c51-546f-4ad2-b36d-ddd215dcb8f7"). InnerVolumeSpecName "kube-api-access-h5vk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:40.641561 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641524    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-token\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-token\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.641745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641576    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-data\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.641745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641611    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-token\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-token\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.641745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641650    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-certs\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.641745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641734    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-socket\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.642013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641796    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-certs\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.642013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641823    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-credential-socket\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.642013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641878    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-envoy\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.642013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641925    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-envoy\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.642013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.641978    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-podinfo\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.642263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642036    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istiod-ca-cert\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.642263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642110    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-socket" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.642263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642211    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-podinfo\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.642263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642244    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istiod-ca-cert\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.642483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642276    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-kube-api-access-46clt\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-kube-api-access-46clt\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.642483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642335    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-credential-socket\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.642483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642400    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-data\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.642483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642434    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-kube-api-access-b9wfx\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-kube-api-access-b9wfx\") pod \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\" (UID: \"c2fdb194-be85-4d72-9df4-fd9ec332ef3f\") "
Oct 29 23:08:40.642685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642506    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-socket\") pod \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\" (UID: \"739f2a27-1668-47cf-9b3e-5beab9d7cf1d\") "
Oct 29 23:08:40.643331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643303    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643335    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643349    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643372    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643385    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643397    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643409    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643420    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643435    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vk5\" (UniqueName: \"kubernetes.io/projected/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-kube-api-access-h5vk5\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.643451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643447    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b649c51-546f-4ad2-b36d-ddd215dcb8f7-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.644370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642245    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-data" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.644370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.642620    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-certs" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.644554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643107    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-data" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.644554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643288    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-credential-socket" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.644554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643472    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-socket" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.644554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.643759    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istiod-ca-cert" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:40.644554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.644336    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istiod-ca-cert" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:40.645514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.645188    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-podinfo" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:40.645514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.645246    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-token" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:40.645514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.645473    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-certs" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.645514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.645485    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-credential-socket" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.647575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.647512    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-kube-api-access-46clt" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "kube-api-access-46clt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:40.647575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.647566    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-token" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:40.649913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.648170    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-kube-api-access-b9wfx" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "kube-api-access-b9wfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:40.649913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.648338    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-podinfo" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:08:40.649913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.648670    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-envoy" pod "739f2a27-1668-47cf-9b3e-5beab9d7cf1d" (UID: "739f2a27-1668-47cf-9b3e-5beab9d7cf1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.649913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.649169    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-envoy" pod "c2fdb194-be85-4d72-9df4-fd9ec332ef3f" (UID: "c2fdb194-be85-4d72-9df4-fd9ec332ef3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:40.744602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744571    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744618    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744633    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744644    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744656    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744667    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744693    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744711    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744723    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744736    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744751    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744813    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744827    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46clt\" (UniqueName: \"kubernetes.io/projected/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-kube-api-access-46clt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744839    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744853    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744865    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9wfx\" (UniqueName: \"kubernetes.io/projected/c2fdb194-be85-4d72-9df4-fd9ec332ef3f-kube-api-access-b9wfx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.744935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.744896    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739f2a27-1668-47cf-9b3e-5beab9d7cf1d-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:40.926143 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.926038    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:40.926991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.926955    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 23:08:40.927158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.927009    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 23:08:40.929996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.929970    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:40.931321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.931302    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 23:08:40.931447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:40.931349    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 23:08:41.224172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.224084    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"
Oct 29 23:08:41.224172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.224104    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf" event={"ID":"6b649c51-546f-4ad2-b36d-ddd215dcb8f7","Type":"ContainerDied","Data":"b2006f2003580ee0d8476d68f9382ecc86c38e1260ff4656737bbf168da4ea50"}
Oct 29 23:08:41.224698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.224177    4827 scope.go:122] "RemoveContainer" containerID="c28f89e53885e6d18f93c9d00a0dc5b0b0dee057a9d307cbf719fad352c0ec30"
Oct 29 23:08:41.226348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.226224    4827 generic.go:356] "Generic (PLEG): container finished" podID="b5fda436-26f7-4579-8456-e804c9f0fb64" containerID="53c4dc275adf6a3778f2cd408e363db605bcfc143c94fc60f96af313705475ce" exitCode=0
Oct 29 23:08:41.226348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.226301    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr" event={"ID":"b5fda436-26f7-4579-8456-e804c9f0fb64","Type":"ContainerDied","Data":"53c4dc275adf6a3778f2cd408e363db605bcfc143c94fc60f96af313705475ce"}
Oct 29 23:08:41.227837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.227822    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"
Oct 29 23:08:41.227961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.227872    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7" event={"ID":"c2fdb194-be85-4d72-9df4-fd9ec332ef3f","Type":"ContainerDied","Data":"f643e61cb68ed9eb6f95b1772f527f1d3f6b0cb87caf0503bd6a2a3b28967d2e"}
Oct 29 23:08:41.230583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.229993    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"
Oct 29 23:08:41.230583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.230008    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq" event={"ID":"739f2a27-1668-47cf-9b3e-5beab9d7cf1d","Type":"ContainerDied","Data":"466f58b59563706d042dcac00f4dde7427e4ad50edcb811668ad1e2c53127a8a"}
Oct 29 23:08:41.236745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.236730    4827 scope.go:122] "RemoveContainer" containerID="0d350dc35f7d683da0453be8fbf55a134d26d16875bff1498d6f68710068473f"
Oct 29 23:08:41.246643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.246629    4827 scope.go:122] "RemoveContainer" containerID="3ebe0e778b340f6b12dd3c7f150e18b3e10238c20b21147fdd7a1596c50e36ea"
Oct 29 23:08:41.331435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.331401    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"]
Oct 29 23:08:41.341980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.341949    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-fcxsf"]
Oct 29 23:08:41.458399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.458371    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"]
Oct 29 23:08:41.484845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.463234    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-jn7n7"]
Oct 29 23:08:41.524012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.523978    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"]
Oct 29 23:08:41.530063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.530039    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-vhhrq"]
Oct 29 23:08:41.924757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.924714    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 23:08:41.925044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.925010    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 23:08:41.930106 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.930082    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 23:08:41.930226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:41.930130    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.199978    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"]
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201058    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201082    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201108    4827 cpu_manager.go:414] "containerMap: removing container" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201116    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201165    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201178    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201215    4827 cpu_manager.go:414] "containerMap: removing container" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201238    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201375    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201391    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy"
Oct 29 23:08:42.204838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.201406    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" containerName="istio-proxy"
Oct 29 23:08:42.237995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.237961    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"]
Oct 29 23:08:42.238645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.238622    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.315314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.315272    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.315605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.315584    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.316016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.315995    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462rl\" (UniqueName: \"kubernetes.io/projected/af01c502-f7a1-4b01-8738-8a0a55676506-kube-api-access-462rl\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.428058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.428016    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462rl\" (UniqueName: \"kubernetes.io/projected/af01c502-f7a1-4b01-8738-8a0a55676506-kube-api-access-462rl\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.428224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.428149    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.428224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.428182    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.432490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.432458    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.450992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.450941    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.455479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.455381    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-462rl\" (UniqueName: \"kubernetes.io/projected/af01c502-f7a1-4b01-8738-8a0a55676506-kube-api-access-462rl\") pod \"maas-api-revoke-sub-d8f68f16-p7n97\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.538142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.538112    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b649c51-546f-4ad2-b36d-ddd215dcb8f7" path="/var/lib/kubelet/pods/6b649c51-546f-4ad2-b36d-ddd215dcb8f7/volumes"
Oct 29 23:08:42.538596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.538583    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" path="/var/lib/kubelet/pods/739f2a27-1668-47cf-9b3e-5beab9d7cf1d/volumes"
Oct 29 23:08:42.539068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.539053    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fdb194-be85-4d72-9df4-fd9ec332ef3f" path="/var/lib/kubelet/pods/c2fdb194-be85-4d72-9df4-fd9ec332ef3f/volumes"
Oct 29 23:08:42.720122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.720031    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:42.835066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.834800    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:42.925430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.925359    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 23:08:42.926070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.925669    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 23:08:42.930316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.930220    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 23:08:42.930316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.930269    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 23:08:42.936575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.936533    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-openshift-ca\") pod \"b5fda436-26f7-4579-8456-e804c9f0fb64\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") "
Oct 29 23:08:42.936698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.936650    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fda436-26f7-4579-8456-e804c9f0fb64-kube-api-access-9xpdq\" (UniqueName: \"kubernetes.io/projected/b5fda436-26f7-4579-8456-e804c9f0fb64-kube-api-access-9xpdq\") pod \"b5fda436-26f7-4579-8456-e804c9f0fb64\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") "
Oct 29 23:08:42.936779 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.936687    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-xks-ca\") pod \"b5fda436-26f7-4579-8456-e804c9f0fb64\" (UID: \"b5fda436-26f7-4579-8456-e804c9f0fb64\") "
Oct 29 23:08:42.936942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.936920    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-openshift-ca" pod "b5fda436-26f7-4579-8456-e804c9f0fb64" (UID: "b5fda436-26f7-4579-8456-e804c9f0fb64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:42.937252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.937232    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:42.938995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.938971    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fda436-26f7-4579-8456-e804c9f0fb64-kube-api-access-9xpdq" pod "b5fda436-26f7-4579-8456-e804c9f0fb64" (UID: "b5fda436-26f7-4579-8456-e804c9f0fb64"). InnerVolumeSpecName "kube-api-access-9xpdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:42.941825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:42.941800    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-xks-ca" pod "b5fda436-26f7-4579-8456-e804c9f0fb64" (UID: "b5fda436-26f7-4579-8456-e804c9f0fb64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:43.038148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.038055    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xpdq\" (UniqueName: \"kubernetes.io/projected/b5fda436-26f7-4579-8456-e804c9f0fb64-kube-api-access-9xpdq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:43.038148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.038084    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fda436-26f7-4579-8456-e804c9f0fb64-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:43.117820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.117738    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"]
Oct 29 23:08:43.118859 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:43.118836    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf01c502_f7a1_4b01_8738_8a0a55676506.slice/crio-469216fa15386f06a7019876b951b1401429c68822358f2e6e254b79275040cd WatchSource:0}: Error finding container 469216fa15386f06a7019876b951b1401429c68822358f2e6e254b79275040cd: Status 404 returned error can't find the container with id 469216fa15386f06a7019876b951b1401429c68822358f2e6e254b79275040cd
Oct 29 23:08:43.400789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.400514    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97" event={"ID":"af01c502-f7a1-4b01-8738-8a0a55676506","Type":"ContainerStarted","Data":"469216fa15386f06a7019876b951b1401429c68822358f2e6e254b79275040cd"}
Oct 29 23:08:43.412793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.406381    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr" event={"ID":"b5fda436-26f7-4579-8456-e804c9f0fb64","Type":"ContainerDied","Data":"1e2e4521119bc15f502829f3595aa7273252b76aca9224abfb97f0f434ab0da9"}
Oct 29 23:08:43.412793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.406436    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e2e4521119bc15f502829f3595aa7273252b76aca9224abfb97f0f434ab0da9"
Oct 29 23:08:43.412793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.406506    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"
Oct 29 23:08:43.924847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.924809    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 23:08:43.925040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.924878    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 23:08:43.930479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.930454    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 23:08:43.930623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:43.930494    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 23:08:44.427323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:44.427292    4827 generic.go:356] "Generic (PLEG): container finished" podID="af01c502-f7a1-4b01-8738-8a0a55676506" containerID="c28595134d4ba87a1a79aa368a2a7a96bf62e8754049e39c6537a589a726617c" exitCode=0
Oct 29 23:08:44.427713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:44.427379    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97" event={"ID":"af01c502-f7a1-4b01-8738-8a0a55676506","Type":"ContainerDied","Data":"c28595134d4ba87a1a79aa368a2a7a96bf62e8754049e39c6537a589a726617c"}
Oct 29 23:08:44.925368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:44.925329    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 23:08:44.925577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:44.925390    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 23:08:44.930136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:44.930104    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 23:08:44.930277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:44.930156    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 23:08:45.704455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.704372    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:45.735739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.735699    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af01c502-f7a1-4b01-8738-8a0a55676506-kube-api-access-462rl\" (UniqueName: \"kubernetes.io/projected/af01c502-f7a1-4b01-8738-8a0a55676506-kube-api-access-462rl\") pod \"af01c502-f7a1-4b01-8738-8a0a55676506\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") "
Oct 29 23:08:45.735951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.735881    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-openshift-ca\") pod \"af01c502-f7a1-4b01-8738-8a0a55676506\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") "
Oct 29 23:08:45.736030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.736009    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-xks-ca\") pod \"af01c502-f7a1-4b01-8738-8a0a55676506\" (UID: \"af01c502-f7a1-4b01-8738-8a0a55676506\") "
Oct 29 23:08:45.736349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.736324    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-openshift-ca" pod "af01c502-f7a1-4b01-8738-8a0a55676506" (UID: "af01c502-f7a1-4b01-8738-8a0a55676506"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:45.738097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.737752    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.740983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.740168    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-xks-ca" pod "af01c502-f7a1-4b01-8738-8a0a55676506" (UID: "af01c502-f7a1-4b01-8738-8a0a55676506"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:45.740983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.740306    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af01c502-f7a1-4b01-8738-8a0a55676506-kube-api-access-462rl" pod "af01c502-f7a1-4b01-8738-8a0a55676506" (UID: "af01c502-f7a1-4b01-8738-8a0a55676506"). InnerVolumeSpecName "kube-api-access-462rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:45.839506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.839464    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-462rl\" (UniqueName: \"kubernetes.io/projected/af01c502-f7a1-4b01-8738-8a0a55676506-kube-api-access-462rl\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.839506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.839493    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af01c502-f7a1-4b01-8738-8a0a55676506-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.924731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.924696    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 23:08:45.924908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.924758    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 23:08:45.930051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.930026    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 23:08:45.930196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:45.930072    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 23:08:46.443531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:46.443496    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97" event={"ID":"af01c502-f7a1-4b01-8738-8a0a55676506","Type":"ContainerDied","Data":"469216fa15386f06a7019876b951b1401429c68822358f2e6e254b79275040cd"}
Oct 29 23:08:46.443740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:46.443559    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469216fa15386f06a7019876b951b1401429c68822358f2e6e254b79275040cd"
Oct 29 23:08:46.443740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:46.443515    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"
Oct 29 23:08:47.095580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:47.095347    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:47.096452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:47.096426    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:47.097297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:47.097278    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:08:47.103604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:47.103577    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:47.104010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:47.103992    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:47.107004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:47.106979    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:08:49.233522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:49.233490    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"]
Oct 29 23:08:49.265238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:49.265209    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"]
Oct 29 23:08:50.015341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.015295    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"]
Oct 29 23:08:50.029166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.029136    4827 cpu_manager.go:414] "containerMap: removing container" podUID="af01c502-f7a1-4b01-8738-8a0a55676506" containerName="revoke-keys"
Oct 29 23:08:50.029364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.029350    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af01c502-f7a1-4b01-8738-8a0a55676506" containerName="revoke-keys"
Oct 29 23:08:50.029493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.029481    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b5fda436-26f7-4579-8456-e804c9f0fb64" containerName="revoke-keys"
Oct 29 23:08:50.029569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.029559    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fda436-26f7-4579-8456-e804c9f0fb64" containerName="revoke-keys"
Oct 29 23:08:50.029903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.029884    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="af01c502-f7a1-4b01-8738-8a0a55676506" containerName="revoke-keys"
Oct 29 23:08:50.030038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.030024    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fda436-26f7-4579-8456-e804c9f0fb64" containerName="revoke-keys"
Oct 29 23:08:50.030112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.030102    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="739f2a27-1668-47cf-9b3e-5beab9d7cf1d" containerName="istio-proxy"
Oct 29 23:08:50.036826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.036610    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.054856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.054820    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:50.056093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.056069    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:08:50.073190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.073160    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.073572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.073545    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.086447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.080464    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpm4b\" (UniqueName: \"kubernetes.io/projected/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-kube-api-access-bpm4b\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.090153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.090123    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"]
Oct 29 23:08:50.182196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.182157    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.182393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.182231    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpm4b\" (UniqueName: \"kubernetes.io/projected/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-kube-api-access-bpm4b\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.182393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.182365    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.183038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.183017    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.184708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.184683    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.207812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.206689    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpm4b\" (UniqueName: \"kubernetes.io/projected/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-kube-api-access-bpm4b\") pod \"maas-api-revoke-sub-d8bd2175-vxl64\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:50.405787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:50.405085    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:52.295847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:52.287899    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:08:52.403797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:52.403049    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 23:08:52.526210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:52.470707    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:08:52.526210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:52.480461    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"]
Oct 29 23:08:52.556859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:52.547794    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:52.736790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:52.704893    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:08:53.199789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.147665    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.201476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.184069    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.202107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.202075    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.305918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.305875    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.306847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.306814    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:08:53.307322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.307303    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:08:53.382017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.379410    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:08:53.412814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.387131    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64" event={"ID":"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02","Type":"ContainerStarted","Data":"89d7597c918b353207f71eff8559b27f2245438019e0a92fbe3e03e062429f79"}
Oct 29 23:08:53.413126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.404868    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.430819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.405512    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.431600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.431572    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:08:53.531875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.447681    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:08:53.531875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.449226    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.531875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.489886    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.531875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.490388    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:08:53.718305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.713879    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:08:53.718305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.714426    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:08:53.840021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.838906    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:53.923109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:53.923080    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:08:54.219958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.207943    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:54.219958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.214648    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:08:54.287748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.227348    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:08:54.287748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.227415    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:08:54.287748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.280348    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk" podUID="5568ec3c-2358-4041-8261-7bb8de202ee0" containerName="payload-pre-processing" containerID="cri-o://0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3" gracePeriod=25
Oct 29 23:08:54.303756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.303716    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk" podUID="be92200b-d421-4f11-9a37-e7c586554482" containerName="payload-processing" containerID="cri-o://4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414" gracePeriod=25
Oct 29 23:08:54.367547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.323958    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:54.367547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.339520    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:08:54.367547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.342158    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:08:54.395909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.390743    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:08:54.395909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.391322    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:08:54.472227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.471304    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:54.472227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.471963    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:54.479462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.472556    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:08:54.479462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.472617    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:08:54.479462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.473502    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:54.503453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.483829    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:08:54.503453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.492727    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:08:54.503453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.493301    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:08:54.665166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.646526    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:08:54.763095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.762865    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64" event={"ID":"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02","Type":"ContainerStarted","Data":"93c8a0117d0a7d7e3cb9842256ab786e450e952f85fbae942cd462f2f70500fb"}
Oct 29 23:08:54.795824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.795797    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"]
Oct 29 23:08:54.796241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.796210    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z" podUID="36b8ec37-28c8-4e00-8810-49379546e146" containerName="maas-api" containerID="cri-o://6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69" gracePeriod=30
Oct 29 23:08:54.799332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.798259    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:08:54.843591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.840149    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:08:54.880391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.873504    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:08:54.880391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.877222    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:08:54.946286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.946258    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:08:54.957480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.957452    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:08:54.962357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.962334    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"]
Oct 29 23:08:54.982531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.982500    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:08:55.000398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.982558    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:08:55.000398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.982589    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:08:55.000398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:54.982673    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.021290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.021161    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"]
Oct 29 23:08:55.053374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.053051    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.053374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.053198    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.053863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.053839    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrf6\" (UniqueName: \"kubernetes.io/projected/f4ddd219-2515-4f10-92fc-cf7398d766bc-kube-api-access-bbrf6\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.091081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.089794    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z" podUID="36b8ec37-28c8-4e00-8810-49379546e146" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.187:8443/health\": dial tcp 10.134.0.187:8443: connect: connection refused"
Oct 29 23:08:55.155945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.155911    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.156070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.156057    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.156278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.156259    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrf6\" (UniqueName: \"kubernetes.io/projected/f4ddd219-2515-4f10-92fc-cf7398d766bc-kube-api-access-bbrf6\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.157942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.157888    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.162940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.162147    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.182162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.182012    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrf6\" (UniqueName: \"kubernetes.io/projected/f4ddd219-2515-4f10-92fc-cf7398d766bc-kube-api-access-bbrf6\") pod \"maas-api-revoke-sub-e0612f85-gqhrl\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.192648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.192545    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:08:55.259067 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.258547    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5568ec3c-2358-4041-8261-7bb8de202ee0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5568ec3c-2358-4041-8261-7bb8de202ee0-plugins-config-volume\") pod \"5568ec3c-2358-4041-8261-7bb8de202ee0\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") "
Oct 29 23:08:55.259067 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.258665    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5568ec3c-2358-4041-8261-7bb8de202ee0-kube-api-access-khqts\" (UniqueName: \"kubernetes.io/projected/5568ec3c-2358-4041-8261-7bb8de202ee0-kube-api-access-khqts\") pod \"5568ec3c-2358-4041-8261-7bb8de202ee0\" (UID: \"5568ec3c-2358-4041-8261-7bb8de202ee0\") "
Oct 29 23:08:55.269131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.269098    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5568ec3c-2358-4041-8261-7bb8de202ee0-plugins-config-volume" pod "5568ec3c-2358-4041-8261-7bb8de202ee0" (UID: "5568ec3c-2358-4041-8261-7bb8de202ee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:55.272611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.272552    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5568ec3c-2358-4041-8261-7bb8de202ee0-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.282992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.282968    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:08:55.295369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.295305    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5568ec3c-2358-4041-8261-7bb8de202ee0-kube-api-access-khqts" pod "5568ec3c-2358-4041-8261-7bb8de202ee0" (UID: "5568ec3c-2358-4041-8261-7bb8de202ee0"). InnerVolumeSpecName "kube-api-access-khqts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:55.297353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.297330    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:08:55.340361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.340320    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:55.375090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.375053    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-maas-api-tls\") pod \"36b8ec37-28c8-4e00-8810-49379546e146\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") "
Oct 29 23:08:55.375506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.375136    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b8ec37-28c8-4e00-8810-49379546e146-kube-api-access-96jtv\" (UniqueName: \"kubernetes.io/projected/36b8ec37-28c8-4e00-8810-49379546e146-kube-api-access-96jtv\") pod \"36b8ec37-28c8-4e00-8810-49379546e146\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") "
Oct 29 23:08:55.375506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.375282    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be92200b-d421-4f11-9a37-e7c586554482-kube-api-access-blrzr\" (UniqueName: \"kubernetes.io/projected/be92200b-d421-4f11-9a37-e7c586554482-kube-api-access-blrzr\") pod \"be92200b-d421-4f11-9a37-e7c586554482\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") "
Oct 29 23:08:55.375506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.375353    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-metrics-certs\") pod \"36b8ec37-28c8-4e00-8810-49379546e146\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") "
Oct 29 23:08:55.375506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.375440    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b8ec37-28c8-4e00-8810-49379546e146-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b8ec37-28c8-4e00-8810-49379546e146-openshift-service-ca\") pod \"36b8ec37-28c8-4e00-8810-49379546e146\" (UID: \"36b8ec37-28c8-4e00-8810-49379546e146\") "
Oct 29 23:08:55.375750 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.375518    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be92200b-d421-4f11-9a37-e7c586554482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92200b-d421-4f11-9a37-e7c586554482-plugins-config-volume\") pod \"be92200b-d421-4f11-9a37-e7c586554482\" (UID: \"be92200b-d421-4f11-9a37-e7c586554482\") "
Oct 29 23:08:55.376451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.376419    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqts\" (UniqueName: \"kubernetes.io/projected/5568ec3c-2358-4041-8261-7bb8de202ee0-kube-api-access-khqts\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.377326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.377282    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be92200b-d421-4f11-9a37-e7c586554482-plugins-config-volume" pod "be92200b-d421-4f11-9a37-e7c586554482" (UID: "be92200b-d421-4f11-9a37-e7c586554482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:55.377804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.377761    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b8ec37-28c8-4e00-8810-49379546e146-openshift-service-ca" pod "36b8ec37-28c8-4e00-8810-49379546e146" (UID: "36b8ec37-28c8-4e00-8810-49379546e146"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:55.378743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.378716    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b8ec37-28c8-4e00-8810-49379546e146-kube-api-access-96jtv" pod "36b8ec37-28c8-4e00-8810-49379546e146" (UID: "36b8ec37-28c8-4e00-8810-49379546e146"). InnerVolumeSpecName "kube-api-access-96jtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:55.379992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.379929    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be92200b-d421-4f11-9a37-e7c586554482-kube-api-access-blrzr" pod "be92200b-d421-4f11-9a37-e7c586554482" (UID: "be92200b-d421-4f11-9a37-e7c586554482"). InnerVolumeSpecName "kube-api-access-blrzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:55.383967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.383921    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-metrics-certs" pod "36b8ec37-28c8-4e00-8810-49379546e146" (UID: "36b8ec37-28c8-4e00-8810-49379546e146"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:55.384067 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.383987    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-maas-api-tls" pod "36b8ec37-28c8-4e00-8810-49379546e146" (UID: "36b8ec37-28c8-4e00-8810-49379546e146"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:55.478304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.478264    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92200b-d421-4f11-9a37-e7c586554482-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.478304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.478307    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.478495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.478324    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96jtv\" (UniqueName: \"kubernetes.io/projected/36b8ec37-28c8-4e00-8810-49379546e146-kube-api-access-96jtv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.478495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.478338    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blrzr\" (UniqueName: \"kubernetes.io/projected/be92200b-d421-4f11-9a37-e7c586554482-kube-api-access-blrzr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.478495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.478353    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b8ec37-28c8-4e00-8810-49379546e146-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.478495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.478367    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b8ec37-28c8-4e00-8810-49379546e146-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:55.606995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.606922    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"]
Oct 29 23:08:55.616139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.616111    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-182195-46njr"]
Oct 29 23:08:55.664050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:55.664011    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"]
Oct 29 23:08:55.669139 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:08:55.669112    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ddd219_2515_4f10_92fc_cf7398d766bc.slice/crio-250aaa04934caf8e9dbe24d8e5467a299ed42ebccda5ca26b7da96284d52d46a WatchSource:0}: Error finding container 250aaa04934caf8e9dbe24d8e5467a299ed42ebccda5ca26b7da96284d52d46a: Status 404 returned error can't find the container with id 250aaa04934caf8e9dbe24d8e5467a299ed42ebccda5ca26b7da96284d52d46a
Oct 29 23:08:56.022863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.021851    4827 generic.go:356] "Generic (PLEG): container finished" podID="be92200b-d421-4f11-9a37-e7c586554482" containerID="4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414" exitCode=0
Oct 29 23:08:56.022863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.022046    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk" event={"ID":"be92200b-d421-4f11-9a37-e7c586554482","Type":"ContainerDied","Data":"4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414"}
Oct 29 23:08:56.022863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.022101    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk" event={"ID":"be92200b-d421-4f11-9a37-e7c586554482","Type":"ContainerDied","Data":"934f76f78c1fbf5548bdee8aff16c62c0b0a38830112461a5dc4db10c611a6c2"}
Oct 29 23:08:56.022863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.022133    4827 scope.go:122] "RemoveContainer" containerID="4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414"
Oct 29 23:08:56.022863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.022217    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"
Oct 29 23:08:56.047974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.047906    4827 generic.go:356] "Generic (PLEG): container finished" podID="5568ec3c-2358-4041-8261-7bb8de202ee0" containerID="0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3" exitCode=0
Oct 29 23:08:56.048061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.048037    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk" event={"ID":"5568ec3c-2358-4041-8261-7bb8de202ee0","Type":"ContainerDied","Data":"0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3"}
Oct 29 23:08:56.048117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.048090    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk" event={"ID":"5568ec3c-2358-4041-8261-7bb8de202ee0","Type":"ContainerDied","Data":"eef155d296ad0e7e73f8df3ba9c3f197f4f3116bc983cf8283d97cb1755a2a35"}
Oct 29 23:08:56.048214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.048194    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"
Oct 29 23:08:56.060049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.055680    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl" event={"ID":"f4ddd219-2515-4f10-92fc-cf7398d766bc","Type":"ContainerStarted","Data":"250aaa04934caf8e9dbe24d8e5467a299ed42ebccda5ca26b7da96284d52d46a"}
Oct 29 23:08:56.079544 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.079517    4827 scope.go:122] "RemoveContainer" containerID="4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414"
Oct 29 23:08:56.083355 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:56.083329    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414\": container with ID starting with 4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414 not found: ID does not exist" containerID="4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414"
Oct 29 23:08:56.083521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.083494    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414"} err="failed to get container status \"4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414\": rpc error: code = NotFound desc = could not find container \"4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414\": container with ID starting with 4f91b57ec1a841a11b31628a7f0c128d3ce36231863234abb5ee6b2719746414 not found: ID does not exist"
Oct 29 23:08:56.083636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.083619    4827 scope.go:122] "RemoveContainer" containerID="0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3"
Oct 29 23:08:56.083744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.083633    4827 generic.go:356] "Generic (PLEG): container finished" podID="f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" containerID="93c8a0117d0a7d7e3cb9842256ab786e450e952f85fbae942cd462f2f70500fb" exitCode=0
Oct 29 23:08:56.083880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.083860    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64" event={"ID":"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02","Type":"ContainerDied","Data":"93c8a0117d0a7d7e3cb9842256ab786e450e952f85fbae942cd462f2f70500fb"}
Oct 29 23:08:56.088066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.087828    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"
Oct 29 23:08:56.088571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.088525    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z" event={"ID":"36b8ec37-28c8-4e00-8810-49379546e146","Type":"ContainerDied","Data":"6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69"}
Oct 29 23:08:56.089038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.088366    4827 generic.go:356] "Generic (PLEG): container finished" podID="36b8ec37-28c8-4e00-8810-49379546e146" containerID="6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69" exitCode=0
Oct 29 23:08:56.089635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.089613    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z" event={"ID":"36b8ec37-28c8-4e00-8810-49379546e146","Type":"ContainerDied","Data":"b0fcb95e716953842c6e82c6e4d0a0ad786e93533cb828ddd02f0fe5d40c1c6e"}
Oct 29 23:08:56.100827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.100083    4827 scope.go:122] "RemoveContainer" containerID="0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3"
Oct 29 23:08:56.100827 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:56.100374    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3\": container with ID starting with 0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3 not found: ID does not exist" containerID="0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3"
Oct 29 23:08:56.100827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.100408    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3"} err="failed to get container status \"0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3\": rpc error: code = NotFound desc = could not find container \"0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3\": container with ID starting with 0a799cbcd048f7efb6ada98a0bf1b91e4057656464bb6b128983a468592b2ac3 not found: ID does not exist"
Oct 29 23:08:56.100827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.100426    4827 scope.go:122] "RemoveContainer" containerID="6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69"
Oct 29 23:08:56.114157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.114062    4827 scope.go:122] "RemoveContainer" containerID="6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69"
Oct 29 23:08:56.115323 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:08:56.114861    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69\": container with ID starting with 6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69 not found: ID does not exist" containerID="6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69"
Oct 29 23:08:56.115323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.114898    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69"} err="failed to get container status \"6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69\": rpc error: code = NotFound desc = could not find container \"6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69\": container with ID starting with 6778c6d69a5c51f69d892bd981652461e369298816447a7ec3cb4b148b571e69 not found: ID does not exist"
Oct 29 23:08:56.115323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.115180    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"]
Oct 29 23:08:56.119170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.119148    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-182195-cdf765db6-ntnxk"]
Oct 29 23:08:56.150374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.150346    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"]
Oct 29 23:08:56.154499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.154472    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-182195-6c6897b5f9-prbxk"]
Oct 29 23:08:56.192706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.192622    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"]
Oct 29 23:08:56.195275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.195248    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-182195-8847cc4c5-n2m7z"]
Oct 29 23:08:56.259060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.259031    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:56.328375 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.328348    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-kube-api-access-bpm4b\" (UniqueName: \"kubernetes.io/projected/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-kube-api-access-bpm4b\") pod \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") "
Oct 29 23:08:56.328516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.328448    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-xks-ca\") pod \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") "
Oct 29 23:08:56.328569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.328555    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-openshift-ca\") pod \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\" (UID: \"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02\") "
Oct 29 23:08:56.328925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.328898    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-openshift-ca" pod "f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" (UID: "f788d17a-65b2-4bb4-bbc9-53c1dd4dea02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:56.329204 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.329186    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.330569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.330540    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-xks-ca" pod "f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" (UID: "f788d17a-65b2-4bb4-bbc9-53c1dd4dea02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:56.330660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.330576    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-kube-api-access-bpm4b" pod "f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" (UID: "f788d17a-65b2-4bb4-bbc9-53c1dd4dea02"). InnerVolumeSpecName "kube-api-access-bpm4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:56.430113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.430081    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpm4b\" (UniqueName: \"kubernetes.io/projected/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-kube-api-access-bpm4b\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.430113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.430107    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.490355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.490280    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b8ec37-28c8-4e00-8810-49379546e146" path="/var/lib/kubelet/pods/36b8ec37-28c8-4e00-8810-49379546e146/volumes"
Oct 29 23:08:56.490809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.490792    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5568ec3c-2358-4041-8261-7bb8de202ee0" path="/var/lib/kubelet/pods/5568ec3c-2358-4041-8261-7bb8de202ee0/volumes"
Oct 29 23:08:56.491233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.491221    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fda436-26f7-4579-8456-e804c9f0fb64" path="/var/lib/kubelet/pods/b5fda436-26f7-4579-8456-e804c9f0fb64/volumes"
Oct 29 23:08:56.491898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:56.491878    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be92200b-d421-4f11-9a37-e7c586554482" path="/var/lib/kubelet/pods/be92200b-d421-4f11-9a37-e7c586554482/volumes"
Oct 29 23:08:57.097990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:57.097913    4827 generic.go:356] "Generic (PLEG): container finished" podID="f4ddd219-2515-4f10-92fc-cf7398d766bc" containerID="ce0b7b972dfccc0c6dc8ee518bb62900e47f7c0c0f2c3a0c7ae79d570d38dca4" exitCode=0
Oct 29 23:08:57.098254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:57.098001    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl" event={"ID":"f4ddd219-2515-4f10-92fc-cf7398d766bc","Type":"ContainerDied","Data":"ce0b7b972dfccc0c6dc8ee518bb62900e47f7c0c0f2c3a0c7ae79d570d38dca4"}
Oct 29 23:08:57.099381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:57.099364    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"
Oct 29 23:08:57.099518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:57.099402    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64" event={"ID":"f788d17a-65b2-4bb4-bbc9-53c1dd4dea02","Type":"ContainerDied","Data":"89d7597c918b353207f71eff8559b27f2245438019e0a92fbe3e03e062429f79"}
Oct 29 23:08:57.099518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:57.099450    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89d7597c918b353207f71eff8559b27f2245438019e0a92fbe3e03e062429f79"
Oct 29 23:08:58.272439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.272413    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:58.356163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.356136    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-xks-ca\") pod \"f4ddd219-2515-4f10-92fc-cf7398d766bc\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") "
Oct 29 23:08:58.356361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.356212    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ddd219-2515-4f10-92fc-cf7398d766bc-kube-api-access-bbrf6\" (UniqueName: \"kubernetes.io/projected/f4ddd219-2515-4f10-92fc-cf7398d766bc-kube-api-access-bbrf6\") pod \"f4ddd219-2515-4f10-92fc-cf7398d766bc\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") "
Oct 29 23:08:58.356361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.356304    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-openshift-ca\") pod \"f4ddd219-2515-4f10-92fc-cf7398d766bc\" (UID: \"f4ddd219-2515-4f10-92fc-cf7398d766bc\") "
Oct 29 23:08:58.356738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.356706    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-openshift-ca" pod "f4ddd219-2515-4f10-92fc-cf7398d766bc" (UID: "f4ddd219-2515-4f10-92fc-cf7398d766bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:58.357078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.357051    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:58.358928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.358900    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ddd219-2515-4f10-92fc-cf7398d766bc-kube-api-access-bbrf6" pod "f4ddd219-2515-4f10-92fc-cf7398d766bc" (UID: "f4ddd219-2515-4f10-92fc-cf7398d766bc"). InnerVolumeSpecName "kube-api-access-bbrf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:58.358928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.358911    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-xks-ca" pod "f4ddd219-2515-4f10-92fc-cf7398d766bc" (UID: "f4ddd219-2515-4f10-92fc-cf7398d766bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:58.458681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.458650    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ddd219-2515-4f10-92fc-cf7398d766bc-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:58.458681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:58.458676    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbrf6\" (UniqueName: \"kubernetes.io/projected/f4ddd219-2515-4f10-92fc-cf7398d766bc-kube-api-access-bbrf6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:08:59.118131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:59.118102    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"
Oct 29 23:08:59.118131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:59.118126    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl" event={"ID":"f4ddd219-2515-4f10-92fc-cf7398d766bc","Type":"ContainerDied","Data":"250aaa04934caf8e9dbe24d8e5467a299ed42ebccda5ca26b7da96284d52d46a"}
Oct 29 23:08:59.118386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:08:59.118179    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="250aaa04934caf8e9dbe24d8e5467a299ed42ebccda5ca26b7da96284d52d46a"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.415590    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"]
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417068    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5568ec3c-2358-4041-8261-7bb8de202ee0" containerName="payload-pre-processing"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417091    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5568ec3c-2358-4041-8261-7bb8de202ee0" containerName="payload-pre-processing"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417109    4827 cpu_manager.go:414] "containerMap: removing container" podUID="be92200b-d421-4f11-9a37-e7c586554482" containerName="payload-processing"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417118    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be92200b-d421-4f11-9a37-e7c586554482" containerName="payload-processing"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417142    4827 cpu_manager.go:414] "containerMap: removing container" podUID="36b8ec37-28c8-4e00-8810-49379546e146" containerName="maas-api"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417150    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b8ec37-28c8-4e00-8810-49379546e146" containerName="maas-api"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417176    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" containerName="revoke-keys"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417183    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" containerName="revoke-keys"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417198    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f4ddd219-2515-4f10-92fc-cf7398d766bc" containerName="revoke-keys"
Oct 29 23:09:01.417313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417205    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ddd219-2515-4f10-92fc-cf7398d766bc" containerName="revoke-keys"
Oct 29 23:09:01.418092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417336    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5568ec3c-2358-4041-8261-7bb8de202ee0" containerName="payload-pre-processing"
Oct 29 23:09:01.418092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417348    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="be92200b-d421-4f11-9a37-e7c586554482" containerName="payload-processing"
Oct 29 23:09:01.418092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417361    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b8ec37-28c8-4e00-8810-49379546e146" containerName="maas-api"
Oct 29 23:09:01.418092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417372    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" containerName="revoke-keys"
Oct 29 23:09:01.418092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.417383    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ddd219-2515-4f10-92fc-cf7398d766bc" containerName="revoke-keys"
Oct 29 23:09:01.421784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.421740    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.454509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.451345    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-af237d-openshift-default-dockercfg-9g5k4\""
Oct 29 23:09:01.456800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.456755    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"]
Oct 29 23:09:01.520837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.520751    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.521206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.521180    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.521391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.521362    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq8th\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-kube-api-access-cq8th\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.521573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.521557    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73447f3-09ac-4b57-8704-e46f8713987d-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.544185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.544147    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.544441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.544422    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.544643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.544627    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.544759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.544743    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73447f3-09ac-4b57-8704-e46f8713987d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.544929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.544912    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.664953    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665043    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665101    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8th\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-kube-api-access-cq8th\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665167    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73447f3-09ac-4b57-8704-e46f8713987d-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665232    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665256    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665339    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665360    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73447f3-09ac-4b57-8704-e46f8713987d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.665794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.665394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.666446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.666377    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.677834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.668153    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.677834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.669335    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.677834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.670222    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73447f3-09ac-4b57-8704-e46f8713987d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.704784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.704047    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.713786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.712203    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.713786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.712646    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73447f3-09ac-4b57-8704-e46f8713987d-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.721789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.716454    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.757827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.757185    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq8th\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-kube-api-access-cq8th\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:01.859826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.858870    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"]
Oct 29 23:09:01.878798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.877213    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:01.895792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.889108    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"]
Oct 29 23:09:01.977722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:01.964723    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4504f1-openshift-default-dockercfg-28rzf\""
Oct 29 23:09:02.102789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.098045    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:02.148242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.148210    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-certs\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.148644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.148615    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bx7\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-kube-api-access-p8bx7\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.148943 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.148922    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.149038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.148962    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-envoy\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.149038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.148995    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istiod-ca-cert\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.149195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.149065    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-credential-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.149195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.149165    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-token\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.149350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.149207    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-podinfo\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.149350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.149250    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-data\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.258901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.254677    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.259246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.259219    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-envoy\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.309967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.255582    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.310986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.309903    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istiod-ca-cert\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.311256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.311238    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-credential-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.311485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.311468    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-token\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.311812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.310626    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istiod-ca-cert\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.312969 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.311607    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-podinfo\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.313164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.313143    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-data\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.313345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.313326    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-certs\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.313532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.313513    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bx7\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-kube-api-access-p8bx7\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.314259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.312282    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-credential-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.314691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.314660    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-data\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.314823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.314674    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-certs\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.315291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.315267    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-envoy\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.319160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.319106    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-podinfo\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.343151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.339187    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-token\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.343151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.339786    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bx7\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-kube-api-access-p8bx7\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.497177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.497148    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:02.589589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.587787    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"]
Oct 29 23:09:02.592232 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:02.592204    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73447f3_09ac_4b57_8704_e46f8713987d.slice/crio-d4c364c795591b1eef34c3d3eb57c47a63fc7622895fd1ac16042174914c85d4 WatchSource:0}: Error finding container d4c364c795591b1eef34c3d3eb57c47a63fc7622895fd1ac16042174914c85d4: Status 404 returned error can't find the container with id d4c364c795591b1eef34c3d3eb57c47a63fc7622895fd1ac16042174914c85d4
Oct 29 23:09:02.595665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.595618    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:02.595811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.595696    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:02.595811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.595728    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:02.806821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.806790    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"]
Oct 29 23:09:02.807893 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:02.807867    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba31fd7_d55b_4305_8218_2d13ba7fa4ff.slice/crio-0ca6392142aa139578750814ec6f9d4ed0f7075d51cb2e7d4204992e93acf1c1 WatchSource:0}: Error finding container 0ca6392142aa139578750814ec6f9d4ed0f7075d51cb2e7d4204992e93acf1c1: Status 404 returned error can't find the container with id 0ca6392142aa139578750814ec6f9d4ed0f7075d51cb2e7d4204992e93acf1c1
Oct 29 23:09:02.810809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.810756    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:02.810937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.810847    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:02.810937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:02.810876    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:02.835450 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:09:02.835411    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:09:03.298848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.276932    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" event={"ID":"b73447f3-09ac-4b57-8704-e46f8713987d","Type":"ContainerStarted","Data":"47f0cbc9ded3d9cba64bf50892d51ded48ac300df358da7ac4bbe35081439408"}
Oct 29 23:09:03.298848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.277047    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" event={"ID":"b73447f3-09ac-4b57-8704-e46f8713987d","Type":"ContainerStarted","Data":"d4c364c795591b1eef34c3d3eb57c47a63fc7622895fd1ac16042174914c85d4"}
Oct 29 23:09:03.298848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.295427    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" event={"ID":"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff","Type":"ContainerStarted","Data":"8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db"}
Oct 29 23:09:03.298848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.295485    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" event={"ID":"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff","Type":"ContainerStarted","Data":"0ca6392142aa139578750814ec6f9d4ed0f7075d51cb2e7d4204992e93acf1c1"}
Oct 29 23:09:03.335316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.335247    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podStartSLOduration=2.3352283529999998 podStartE2EDuration="2.335228353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:03.330555276 +0000 UTC m=+1578.953466296" watchObservedRunningTime="2026-10-29 23:09:03.335228353 +0000 UTC m=+1578.958139374"
Oct 29 23:09:03.502431 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.502223    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:03.505460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.503054    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 23:09:03.505739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.505695    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 23:09:03.519079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.519013    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podStartSLOduration=2.518991065 podStartE2EDuration="2.518991065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:03.421494603 +0000 UTC m=+1579.044405626" watchObservedRunningTime="2026-10-29 23:09:03.518991065 +0000 UTC m=+1579.141902086"
Oct 29 23:09:03.526376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.526344    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"]
Oct 29 23:09:03.526707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:03.526680    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" containerID="cri-o://6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e" gracePeriod=30
Oct 29 23:09:04.057898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.057864    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"]
Oct 29 23:09:04.062351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.062334    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.078031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.077926    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:09:04.078264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.078156    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:09:04.082565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.082537    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"]
Oct 29 23:09:04.117748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.117723    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:04.124668 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.124643    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:04.223730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.223697    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.223900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.223879    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4hr\" (UniqueName: \"kubernetes.io/projected/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-kube-api-access-9g4hr\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.224029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.224014    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.303111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.303082    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:04.303927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.303908    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:09:04.325004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.324945    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.325115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.325031    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4hr\" (UniqueName: \"kubernetes.io/projected/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-kube-api-access-9g4hr\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.325115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.325104    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.325567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.325545    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.327418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.327396    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.348032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.348006    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4hr\" (UniqueName: \"kubernetes.io/projected/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-kube-api-access-9g4hr\") pod \"maas-api-revoke-sub-651f398e-frbl8\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.376045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.376008    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:04.506417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.506385    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:04.687249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:04.687208    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"]
Oct 29 23:09:04.689394 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:04.689364    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63da891d_f1e0_4cdc_b9ce_1b8212f49b01.slice/crio-cd0d458489d74817e6aa31307b863f840f1e0f6ea1eaa0720cc3599c4a982c3f WatchSource:0}: Error finding container cd0d458489d74817e6aa31307b863f840f1e0f6ea1eaa0720cc3599c4a982c3f: Status 404 returned error can't find the container with id cd0d458489d74817e6aa31307b863f840f1e0f6ea1eaa0720cc3599c4a982c3f
Oct 29 23:09:05.310511 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:05.310475    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8" event={"ID":"63da891d-f1e0-4cdc-b9ce-1b8212f49b01","Type":"ContainerStarted","Data":"e9fe8b31b25512b326c6ac8acb1bf801df41e039bae3426e8bdfb52fc14f070e"}
Oct 29 23:09:05.310722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:05.310559    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8" event={"ID":"63da891d-f1e0-4cdc-b9ce-1b8212f49b01","Type":"ContainerStarted","Data":"cd0d458489d74817e6aa31307b863f840f1e0f6ea1eaa0720cc3599c4a982c3f"}
Oct 29 23:09:05.311040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:05.311015    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:05.311907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:05.311887    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:09:05.355184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:05.355102    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8" podStartSLOduration=1.355088869 podStartE2EDuration="1.355088869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:05.350049091 +0000 UTC m=+1580.972960124" watchObservedRunningTime="2026-10-29 23:09:05.355088869 +0000 UTC m=+1580.977999889"
Oct 29 23:09:06.318991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:06.318954    4827 generic.go:356] "Generic (PLEG): container finished" podID="63da891d-f1e0-4cdc-b9ce-1b8212f49b01" containerID="e9fe8b31b25512b326c6ac8acb1bf801df41e039bae3426e8bdfb52fc14f070e" exitCode=0
Oct 29 23:09:06.319456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:06.319033    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8" event={"ID":"63da891d-f1e0-4cdc-b9ce-1b8212f49b01","Type":"ContainerDied","Data":"e9fe8b31b25512b326c6ac8acb1bf801df41e039bae3426e8bdfb52fc14f070e"}
Oct 29 23:09:07.485366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.485343    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:07.583382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.583299    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-xks-ca\") pod \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") "
Oct 29 23:09:07.583539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.583395    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-openshift-ca\") pod \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") "
Oct 29 23:09:07.583539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.583419    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-kube-api-access-9g4hr\" (UniqueName: \"kubernetes.io/projected/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-kube-api-access-9g4hr\") pod \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\" (UID: \"63da891d-f1e0-4cdc-b9ce-1b8212f49b01\") "
Oct 29 23:09:07.583745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.583721    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-openshift-ca" pod "63da891d-f1e0-4cdc-b9ce-1b8212f49b01" (UID: "63da891d-f1e0-4cdc-b9ce-1b8212f49b01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:07.584138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.584117    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:07.585510 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.585487    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-xks-ca" pod "63da891d-f1e0-4cdc-b9ce-1b8212f49b01" (UID: "63da891d-f1e0-4cdc-b9ce-1b8212f49b01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:07.585616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.585591    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-kube-api-access-9g4hr" pod "63da891d-f1e0-4cdc-b9ce-1b8212f49b01" (UID: "63da891d-f1e0-4cdc-b9ce-1b8212f49b01"). InnerVolumeSpecName "kube-api-access-9g4hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:07.613321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.613296    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"]
Oct 29 23:09:07.613886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.613872    4827 cpu_manager.go:414] "containerMap: removing container" podUID="63da891d-f1e0-4cdc-b9ce-1b8212f49b01" containerName="revoke-keys"
Oct 29 23:09:07.613933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.613887    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63da891d-f1e0-4cdc-b9ce-1b8212f49b01" containerName="revoke-keys"
Oct 29 23:09:07.613991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.613982    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="63da891d-f1e0-4cdc-b9ce-1b8212f49b01" containerName="revoke-keys"
Oct 29 23:09:07.617384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.617366    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.637229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.637182    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-32717a-openshift-default-dockercfg-4th7d\""
Oct 29 23:09:07.643399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.643372    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"]
Oct 29 23:09:07.685891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.685862    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhkm\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-kube-api-access-7zhkm\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.685925    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686060    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686197    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686233    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686369    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686403    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b66c029-ab0c-4a37-82fc-889fb3667b97-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686509    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686561    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.686736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686721    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4hr\" (UniqueName: \"kubernetes.io/projected/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-kube-api-access-9g4hr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:07.686792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.686746    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da891d-f1e0-4cdc-b9ce-1b8212f49b01-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788349    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhkm\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-kube-api-access-7zhkm\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788442    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788504    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788601    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788628    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788719    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788753    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b66c029-ab0c-4a37-82fc-889fb3667b97-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788838    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.788896    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.789567    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.789831    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.795058    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b66c029-ab0c-4a37-82fc-889fb3667b97-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.795298    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.809791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.809581    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.820793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.810912    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.820793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.815582    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhkm\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-kube-api-access-7zhkm\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.847853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.845020    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.867761 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.866318    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:07.940784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:07.940495    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:08.333800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.333737    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"
Oct 29 23:09:08.333963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.333803    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8" event={"ID":"63da891d-f1e0-4cdc-b9ce-1b8212f49b01","Type":"ContainerDied","Data":"cd0d458489d74817e6aa31307b863f840f1e0f6ea1eaa0720cc3599c4a982c3f"}
Oct 29 23:09:08.333963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.333884    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd0d458489d74817e6aa31307b863f840f1e0f6ea1eaa0720cc3599c4a982c3f"
Oct 29 23:09:08.458912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.458877    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"]
Oct 29 23:09:08.459326 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:08.459304    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b66c029_ab0c_4a37_82fc_889fb3667b97.slice/crio-7cb7a47e0eb518f1d102bc2499a7c781e813725a89b55914db4accbe491c342f WatchSource:0}: Error finding container 7cb7a47e0eb518f1d102bc2499a7c781e813725a89b55914db4accbe491c342f: Status 404 returned error can't find the container with id 7cb7a47e0eb518f1d102bc2499a7c781e813725a89b55914db4accbe491c342f
Oct 29 23:09:08.461941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.461906    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:08.462033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.461972    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:08.462033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.462021    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:08.511899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.511869    4827 patch_prober.go:29] interesting pod/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:09:08.512247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.511914    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:09:08.611913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.611880    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"]
Oct 29 23:09:08.612304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:08.612267    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl" podUID="30674ff2-2aeb-4768-9611-d61ea2e7eb80" containerName="discovery" containerID="cri-o://c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8" gracePeriod=30
Oct 29 23:09:09.045912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.045884    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:09:09.079925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.079898    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:09:09.144234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.144196    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-socket\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.144442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.144382    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-ca-configmap\") pod \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") "
Oct 29 23:09:09.144572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.144552    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-data\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.144653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.144604    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-podinfo\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.151953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.148580    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-ca-configmap" pod "30674ff2-2aeb-4768-9611-d61ea2e7eb80" (UID: "30674ff2-2aeb-4768-9611-d61ea2e7eb80"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:09.151953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.148949    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-socket" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:09.151953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.149220    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-data" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:09.152897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.152872    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-credential-socket\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.153146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.152946    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-dns-cert\") pod \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") "
Oct 29 23:09:09.153146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.152982    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-certs\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.153146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153038    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-kubeconfig\") pod \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") "
Oct 29 23:09:09.153146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153066    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-envoy\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.153146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153135    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-cacerts\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-cacerts\") pod \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") "
Oct 29 23:09:09.153400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153166    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-token\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-token\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.153400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153208    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-token\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-token\") pod \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") "
Oct 29 23:09:09.153400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153247    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30674ff2-2aeb-4768-9611-d61ea2e7eb80-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30674ff2-2aeb-4768-9611-d61ea2e7eb80-local-certs\") pod \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") "
Oct 29 23:09:09.153400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153277    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-kube-api-access-zw9dg\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-kube-api-access-zw9dg\") pod \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\" (UID: \"30674ff2-2aeb-4768-9611-d61ea2e7eb80\") "
Oct 29 23:09:09.153400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153351    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-kube-api-access-bst2h\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-kube-api-access-bst2h\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.153571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.153405    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istiod-ca-cert\") pod \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\" (UID: \"c98f78c6-6603-4c29-9b23-721bacf6d8d8\") "
Oct 29 23:09:09.154230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.154199    4827 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-ca-configmap\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.154230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.154210    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-podinfo" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:09:09.154230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.154230    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.154575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.154245    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.154633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.154569    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istiod-ca-cert" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:09.155707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.155619    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-credential-socket" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:09.155861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.155738    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-certs" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:09.158189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.158014    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-kubeconfig" pod "30674ff2-2aeb-4768-9611-d61ea2e7eb80" (UID: "30674ff2-2aeb-4768-9611-d61ea2e7eb80"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:09.158424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.158393    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-dns-cert" pod "30674ff2-2aeb-4768-9611-d61ea2e7eb80" (UID: "30674ff2-2aeb-4768-9611-d61ea2e7eb80"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:09.159078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.159058    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30674ff2-2aeb-4768-9611-d61ea2e7eb80-local-certs" pod "30674ff2-2aeb-4768-9611-d61ea2e7eb80" (UID: "30674ff2-2aeb-4768-9611-d61ea2e7eb80"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:09.159179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.159082    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-kube-api-access-zw9dg" pod "30674ff2-2aeb-4768-9611-d61ea2e7eb80" (UID: "30674ff2-2aeb-4768-9611-d61ea2e7eb80"). InnerVolumeSpecName "kube-api-access-zw9dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:09.159179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.159114    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-token" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:09.159179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.159161    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-cacerts" pod "30674ff2-2aeb-4768-9611-d61ea2e7eb80" (UID: "30674ff2-2aeb-4768-9611-d61ea2e7eb80"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:09.159543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.159525    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-token" pod "30674ff2-2aeb-4768-9611-d61ea2e7eb80" (UID: "30674ff2-2aeb-4768-9611-d61ea2e7eb80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:09.160135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.160113    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-kube-api-access-bst2h" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "kube-api-access-bst2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:09.160523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.160503    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-envoy" pod "c98f78c6-6603-4c29-9b23-721bacf6d8d8" (UID: "c98f78c6-6603-4c29-9b23-721bacf6d8d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:09.255903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255827    4827 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-kubeconfig\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.255903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255855    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.255903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255867    4827 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-cacerts\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.255903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255878    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.255903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255891    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.255903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255902    4827 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30674ff2-2aeb-4768-9611-d61ea2e7eb80-local-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.256211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255917    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw9dg\" (UniqueName: \"kubernetes.io/projected/30674ff2-2aeb-4768-9611-d61ea2e7eb80-kube-api-access-zw9dg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.256211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255931    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bst2h\" (UniqueName: \"kubernetes.io/projected/c98f78c6-6603-4c29-9b23-721bacf6d8d8-kube-api-access-bst2h\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.256211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255942    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.256211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255953    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f78c6-6603-4c29-9b23-721bacf6d8d8-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.256211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255964    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.256211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255976    4827 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30674ff2-2aeb-4768-9611-d61ea2e7eb80-istio-csr-dns-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.256211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.255987    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f78c6-6603-4c29-9b23-721bacf6d8d8-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:09.259279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259251    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"]
Oct 29 23:09:09.259849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259835    4827 cpu_manager.go:414] "containerMap: removing container" podUID="30674ff2-2aeb-4768-9611-d61ea2e7eb80" containerName="discovery"
Oct 29 23:09:09.259904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259851    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30674ff2-2aeb-4768-9611-d61ea2e7eb80" containerName="discovery"
Oct 29 23:09:09.259904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259866    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy"
Oct 29 23:09:09.259904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259871    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy"
Oct 29 23:09:09.259998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259975    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy"
Oct 29 23:09:09.259998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259988    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy"
Oct 29 23:09:09.260079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.259999    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="30674ff2-2aeb-4768-9611-d61ea2e7eb80" containerName="discovery"
Oct 29 23:09:09.264618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.264596    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.282041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.282011    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-af237d\""
Oct 29 23:09:09.282597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.282575    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-af237d-dockercfg-zwwlw\""
Oct 29 23:09:09.283446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.283405    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"]
Oct 29 23:09:09.410602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.393814    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"]
Oct 29 23:09:09.410602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.394692    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy"
Oct 29 23:09:09.410602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.394712    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerName="istio-proxy"
Oct 29 23:09:09.436809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.436063    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:09.515048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.514951    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"]
Oct 29 23:09:09.544355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.544313    4827 generic.go:356] "Generic (PLEG): container finished" podID="30674ff2-2aeb-4768-9611-d61ea2e7eb80" containerID="c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8" exitCode=0
Oct 29 23:09:09.544563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.544538    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl" event={"ID":"30674ff2-2aeb-4768-9611-d61ea2e7eb80","Type":"ContainerDied","Data":"c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8"}
Oct 29 23:09:09.544644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.544601    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl" event={"ID":"30674ff2-2aeb-4768-9611-d61ea2e7eb80","Type":"ContainerDied","Data":"6b4f47bf081ded4537c08697fc321452579f3294dab5fbeb6777c1af531f3534"}
Oct 29 23:09:09.544644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.544630    4827 scope.go:122] "RemoveContainer" containerID="c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8"
Oct 29 23:09:09.550376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.544831    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"
Oct 29 23:09:09.598141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.579757    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca901e7-95bf-4acf-89ad-4d06c2e616fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.598141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.580050    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqr4\" (UniqueName: \"kubernetes.io/projected/eca901e7-95bf-4acf-89ad-4d06c2e616fa-kube-api-access-qrqr4\") pod \"payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.598141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.596382    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" event={"ID":"8b66c029-ab0c-4a37-82fc-889fb3667b97","Type":"ContainerStarted","Data":"2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c"}
Oct 29 23:09:09.598141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.596450    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" event={"ID":"8b66c029-ab0c-4a37-82fc-889fb3667b97","Type":"ContainerStarted","Data":"7cb7a47e0eb518f1d102bc2499a7c781e813725a89b55914db4accbe491c342f"}
Oct 29 23:09:09.615794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.614606    4827 generic.go:356] "Generic (PLEG): container finished" podID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" containerID="6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e" exitCode=0
Oct 29 23:09:09.615794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.614780    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" event={"ID":"c98f78c6-6603-4c29-9b23-721bacf6d8d8","Type":"ContainerDied","Data":"6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e"}
Oct 29 23:09:09.615794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.614835    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz" event={"ID":"c98f78c6-6603-4c29-9b23-721bacf6d8d8","Type":"ContainerDied","Data":"d278260aec771fdb244de7dfd1ec6ce11a96031a9f09adee72ce601908efb424"}
Oct 29 23:09:09.615794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.614924    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"
Oct 29 23:09:09.628282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.628250    4827 scope.go:122] "RemoveContainer" containerID="c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8"
Oct 29 23:09:09.628753 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:09:09.628722    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8\": container with ID starting with c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8 not found: ID does not exist" containerID="c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8"
Oct 29 23:09:09.628878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.628781    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8"} err="failed to get container status \"c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8\": rpc error: code = NotFound desc = could not find container \"c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8\": container with ID starting with c447972eff04b739ac58ccf315a450a676b99c4e4922f05f6bb845f9feefa6d8 not found: ID does not exist"
Oct 29 23:09:09.628878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.628807    4827 scope.go:122] "RemoveContainer" containerID="6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e"
Oct 29 23:09:09.645802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.645726    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podStartSLOduration=2.64570884 podStartE2EDuration="2.64570884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:09.643001585 +0000 UTC m=+1585.265912605" watchObservedRunningTime="2026-10-29 23:09:09.64570884 +0000 UTC m=+1585.268619873"
Oct 29 23:09:09.653215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.653194    4827 scope.go:122] "RemoveContainer" containerID="7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca"
Oct 29 23:09:09.663100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.663082    4827 scope.go:122] "RemoveContainer" containerID="6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e"
Oct 29 23:09:09.663382 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:09:09.663355    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e\": container with ID starting with 6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e not found: ID does not exist" containerID="6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e"
Oct 29 23:09:09.663432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.663384    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e"} err="failed to get container status \"6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e\": rpc error: code = NotFound desc = could not find container \"6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e\": container with ID starting with 6e90bd6a927bc0e46ae0fdac30fd81e86849cc4cfff6efa7c29abc5018269f8e not found: ID does not exist"
Oct 29 23:09:09.663432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.663400    4827 scope.go:122] "RemoveContainer" containerID="7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca"
Oct 29 23:09:09.663649 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:09:09.663624    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca\": container with ID starting with 7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca not found: ID does not exist" containerID="7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca"
Oct 29 23:09:09.663709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.663644    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca"} err="failed to get container status \"7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca\": rpc error: code = NotFound desc = could not find container \"7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca\": container with ID starting with 7cabee5dc5f5d88838aaa8f88b1f281e6107da65fa1451ea9ba59e75454385ca not found: ID does not exist"
Oct 29 23:09:09.681169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.681147    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"]
Oct 29 23:09:09.681453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.681432    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvh6\" (UniqueName: \"kubernetes.io/projected/440801e3-e740-4278-bcdb-0bd4418b9d2e-kube-api-access-xfvh6\") pod \"payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:09.681615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.681595    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqr4\" (UniqueName: \"kubernetes.io/projected/eca901e7-95bf-4acf-89ad-4d06c2e616fa-kube-api-access-qrqr4\") pod \"payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.681732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.681630    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440801e3-e740-4278-bcdb-0bd4418b9d2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:09.681866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.681840    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca901e7-95bf-4acf-89ad-4d06c2e616fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.682569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.682503    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca901e7-95bf-4acf-89ad-4d06c2e616fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.686617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.686595    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vxstl"]
Oct 29 23:09:09.704881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.704857    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqr4\" (UniqueName: \"kubernetes.io/projected/eca901e7-95bf-4acf-89ad-4d06c2e616fa-kube-api-access-qrqr4\") pod \"payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.720190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.720166    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"]
Oct 29 23:09:09.724327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.724306    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-182195-openshift-default-67d7d87fb5-4j4bz"]
Oct 29 23:09:09.784994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.784803    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvh6\" (UniqueName: \"kubernetes.io/projected/440801e3-e740-4278-bcdb-0bd4418b9d2e-kube-api-access-xfvh6\") pod \"payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:09.785287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.784996    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440801e3-e740-4278-bcdb-0bd4418b9d2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:09.808791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.786077    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440801e3-e740-4278-bcdb-0bd4418b9d2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:09.869789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.866754    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvh6\" (UniqueName: \"kubernetes.io/projected/440801e3-e740-4278-bcdb-0bd4418b9d2e-kube-api-access-xfvh6\") pod \"payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:09.923796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.909903    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:09.948741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.943184    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:09.964219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.963872    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 23:09:09.964219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:09.963929    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 23:09:10.052378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.049557    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"]
Oct 29 23:09:10.063191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.063163    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"]
Oct 29 23:09:10.063371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.063260    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.097412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.097282    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:10.103809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.102357    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"]
Oct 29 23:09:10.109561 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.108001    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.130509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.130480    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-4504f1-dockercfg-2brn5\""
Oct 29 23:09:10.171796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.149474    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"]
Oct 29 23:09:10.171796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.163604    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-4504f1\""
Oct 29 23:09:10.196819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.173500    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrl9\" (UniqueName: \"kubernetes.io/projected/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-kube-api-access-lxrl9\") pod \"payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.196819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.173568    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.196819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.173641    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqhr\" (UniqueName: \"kubernetes.io/projected/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-kube-api-access-shqhr\") pod \"payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.196819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.173672    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.315925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.313285    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrl9\" (UniqueName: \"kubernetes.io/projected/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-kube-api-access-lxrl9\") pod \"payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.315925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.313360    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.315925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.313426    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqhr\" (UniqueName: \"kubernetes.io/projected/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-kube-api-access-shqhr\") pod \"payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.315925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.313457    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.315925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.314442    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.315925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.315539    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.370177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.360837    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"]
Oct 29 23:09:10.370177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.361088    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy" containerID="cri-o://9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58" gracePeriod=30
Oct 29 23:09:10.379720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.379689    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"]
Oct 29 23:09:10.380029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.380002    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" containerID="cri-o://32e5f97ba36284070ba3c43217d15d7f27b35f9da4c3ca87a1aa187733dc872a" gracePeriod=30
Oct 29 23:09:10.389233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.388725    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"]
Oct 29 23:09:10.447386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.446378    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.501475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.501441    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:09:10.502389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.502344    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:09:10.504538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.504516    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqhr\" (UniqueName: \"kubernetes.io/projected/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-kube-api-access-shqhr\") pod \"payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.519369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.519340    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrl9\" (UniqueName: \"kubernetes.io/projected/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-kube-api-access-lxrl9\") pod \"payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.526546 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.526519    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30674ff2-2aeb-4768-9611-d61ea2e7eb80" path="/var/lib/kubelet/pods/30674ff2-2aeb-4768-9611-d61ea2e7eb80/volumes"
Oct 29 23:09:10.534291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.534267    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98f78c6-6603-4c29-9b23-721bacf6d8d8" path="/var/lib/kubelet/pods/c98f78c6-6603-4c29-9b23-721bacf6d8d8/volumes"
Oct 29 23:09:10.539342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.539319    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"]
Oct 29 23:09:10.539556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.539542    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"]
Oct 29 23:09:10.540390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.540365    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.540618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.540602    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.540940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.540917    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z56c\" (UniqueName: \"kubernetes.io/projected/f8341e77-16de-40ab-be80-82277155c7ee-kube-api-access-4z56c\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.541752 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:10.541732    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca901e7_95bf_4acf_89ad_4d06c2e616fa.slice/crio-8f4deada4db3d4621285ee4241f6688c9736c196275a25b90f5a36dab7dcc76b WatchSource:0}: Error finding container 8f4deada4db3d4621285ee4241f6688c9736c196275a25b90f5a36dab7dcc76b: Status 404 returned error can't find the container with id 8f4deada4db3d4621285ee4241f6688c9736c196275a25b90f5a36dab7dcc76b
Oct 29 23:09:10.626134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.626101    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" event={"ID":"eca901e7-95bf-4acf-89ad-4d06c2e616fa","Type":"ContainerStarted","Data":"8f4deada4db3d4621285ee4241f6688c9736c196275a25b90f5a36dab7dcc76b"}
Oct 29 23:09:10.642298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.642268    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.642397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.642305    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z56c\" (UniqueName: \"kubernetes.io/projected/f8341e77-16de-40ab-be80-82277155c7ee-kube-api-access-4z56c\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.642620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.642601    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.643212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.643195    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.644587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.644565    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.659319 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:10.659296    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440801e3_e740_4278_bcdb_0bd4418b9d2e.slice/crio-6bef23f3ed121be78539ced5183c0dc37277fe2453379e61a88e76b82406f7a6 WatchSource:0}: Error finding container 6bef23f3ed121be78539ced5183c0dc37277fe2453379e61a88e76b82406f7a6: Status 404 returned error can't find the container with id 6bef23f3ed121be78539ced5183c0dc37277fe2453379e61a88e76b82406f7a6
Oct 29 23:09:10.659728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.659706    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"]
Oct 29 23:09:10.666304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.666279    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z56c\" (UniqueName: \"kubernetes.io/projected/f8341e77-16de-40ab-be80-82277155c7ee-kube-api-access-4z56c\") pod \"maas-api-revoke-sub-91b0fc18-tht9b\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.687575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.687558    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:10.726628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.726605    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:10.895190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.895161    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:10.947896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:10.947868    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:11.130880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.130851    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"]
Oct 29 23:09:11.146813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.146733    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"]
Oct 29 23:09:11.211355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.211331    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"]
Oct 29 23:09:11.211638 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:11.211614    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8341e77_16de_40ab_be80_82277155c7ee.slice/crio-7abc5b6702be61f0eebe748d6c1927d99e8128b4f4a349213641f86d037a9884 WatchSource:0}: Error finding container 7abc5b6702be61f0eebe748d6c1927d99e8128b4f4a349213641f86d037a9884: Status 404 returned error can't find the container with id 7abc5b6702be61f0eebe748d6c1927d99e8128b4f4a349213641f86d037a9884
Oct 29 23:09:11.634646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.634610    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" event={"ID":"afa6dcd1-b9cb-41e9-bc66-4bdc29478492","Type":"ContainerStarted","Data":"d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780"}
Oct 29 23:09:11.635151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.634681    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:11.635151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.634707    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" event={"ID":"afa6dcd1-b9cb-41e9-bc66-4bdc29478492","Type":"ContainerStarted","Data":"2d2affb8d05d0b3cfacb8ad7e0ee6657f6279c208e24eeb9277c308f5205cba1"}
Oct 29 23:09:11.636530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.636495    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" event={"ID":"fd4199f4-e699-464a-9d1d-5c23ba2cc72e","Type":"ContainerStarted","Data":"b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf"}
Oct 29 23:09:11.636671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.636541    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" event={"ID":"fd4199f4-e699-464a-9d1d-5c23ba2cc72e","Type":"ContainerStarted","Data":"1869fc6fe1b2f5d419c1483d7818dac8725f40826bd652e172fa398d7b5d4d19"}
Oct 29 23:09:11.636671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.636568    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:11.638159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.638132    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" event={"ID":"eca901e7-95bf-4acf-89ad-4d06c2e616fa","Type":"ContainerStarted","Data":"54b37c43f28e6318d3cf97d2651e6b7c402e08a81a101445187d6e2a0d2fb892"}
Oct 29 23:09:11.638272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.638188    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:11.639801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.639740    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" event={"ID":"440801e3-e740-4278-bcdb-0bd4418b9d2e","Type":"ContainerStarted","Data":"864a0e6c3d7988b857bd39e7986b4d272f035b806ccfc41f2284229134286628"}
Oct 29 23:09:11.639910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.639816    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" event={"ID":"440801e3-e740-4278-bcdb-0bd4418b9d2e","Type":"ContainerStarted","Data":"6bef23f3ed121be78539ced5183c0dc37277fe2453379e61a88e76b82406f7a6"}
Oct 29 23:09:11.639986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.639935    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:11.641404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.641380    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b" event={"ID":"f8341e77-16de-40ab-be80-82277155c7ee","Type":"ContainerStarted","Data":"f8a3b44a761fdef5e36db6e2a6c574bef3531ea2630d893656a147c94fcc6fb0"}
Oct 29 23:09:11.641508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.641422    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b" event={"ID":"f8341e77-16de-40ab-be80-82277155c7ee","Type":"ContainerStarted","Data":"7abc5b6702be61f0eebe748d6c1927d99e8128b4f4a349213641f86d037a9884"}
Oct 29 23:09:11.641584 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.641569    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:11.642435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.642416    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:09:11.699788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.678868    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" podStartSLOduration=1.678848476 podStartE2EDuration="1.678848476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:11.674683653 +0000 UTC m=+1587.297594672" watchObservedRunningTime="2026-10-29 23:09:11.678848476 +0000 UTC m=+1587.301759499"
Oct 29 23:09:11.728852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.726943    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" podStartSLOduration=1.726925209 podStartE2EDuration="1.726925209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:11.724916547 +0000 UTC m=+1587.347827567" watchObservedRunningTime="2026-10-29 23:09:11.726925209 +0000 UTC m=+1587.349836232"
Oct 29 23:09:11.766744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.766690    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" podStartSLOduration=2.76667555 podStartE2EDuration="2.76667555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:11.764818202 +0000 UTC m=+1587.387729239" watchObservedRunningTime="2026-10-29 23:09:11.76667555 +0000 UTC m=+1587.389586612"
Oct 29 23:09:11.847504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.847435    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b" podStartSLOduration=1.847415814 podStartE2EDuration="1.847415814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:11.805290317 +0000 UTC m=+1587.428201337" watchObservedRunningTime="2026-10-29 23:09:11.847415814 +0000 UTC m=+1587.470326835"
Oct 29 23:09:11.848470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:11.848441    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" podStartSLOduration=2.84843247 podStartE2EDuration="2.84843247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:11.845268938 +0000 UTC m=+1587.468179958" watchObservedRunningTime="2026-10-29 23:09:11.84843247 +0000 UTC m=+1587.471343490"
Oct 29 23:09:12.650214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.650176    4827 generic.go:356] "Generic (PLEG): container finished" podID="f8341e77-16de-40ab-be80-82277155c7ee" containerID="f8a3b44a761fdef5e36db6e2a6c574bef3531ea2630d893656a147c94fcc6fb0" exitCode=0
Oct 29 23:09:12.650615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.650256    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b" event={"ID":"f8341e77-16de-40ab-be80-82277155c7ee","Type":"ContainerDied","Data":"f8a3b44a761fdef5e36db6e2a6c574bef3531ea2630d893656a147c94fcc6fb0"}
Oct 29 23:09:12.944167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.944085    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"]
Oct 29 23:09:12.952110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.952082    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979232    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979281    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979321    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979355    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979386    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979489    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979547    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpslq\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-kube-api-access-tpslq\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979604    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:12.979702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:12.979692    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.002464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.002432    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"]
Oct 29 23:09:13.020102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.020074    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a9ced3-openshift-default-dockercfg-mhf8n\""
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.089284    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.089340    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.089455    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.089499    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpslq\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-kube-api-access-tpslq\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.089553    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.089629    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.089749    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.090212    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.091317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.090460    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.117944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.092594    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.117944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.092834    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.117944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.110462    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.118396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.118366    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.118589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.118567    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.125785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.119905    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.129022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.128997    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.145802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.145109    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.145965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.145805    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpslq\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-kube-api-access-tpslq\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.281839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.272063    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:13.759849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.759818    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"]
Oct 29 23:09:13.760958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.760930    4827 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:09:13.761184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.761149    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:09:13.763190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.763161    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:13.763261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.763225    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:13.763261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.763253    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:13.818474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.818448    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:13.917835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.917191    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"]
Oct 29 23:09:13.922789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.918220    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f8341e77-16de-40ab-be80-82277155c7ee" containerName="revoke-keys"
Oct 29 23:09:13.922789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.918245    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8341e77-16de-40ab-be80-82277155c7ee" containerName="revoke-keys"
Oct 29 23:09:13.922789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.918402    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8341e77-16de-40ab-be80-82277155c7ee" containerName="revoke-keys"
Oct 29 23:09:13.923489 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.923455    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:13.940583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.940550    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-32717a\""
Oct 29 23:09:13.940813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.940794    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-32717a-dockercfg-lzdms\""
Oct 29 23:09:13.942463 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.942438    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8341e77-16de-40ab-be80-82277155c7ee-kube-api-access-4z56c\" (UniqueName: \"kubernetes.io/projected/f8341e77-16de-40ab-be80-82277155c7ee-kube-api-access-4z56c\") pod \"f8341e77-16de-40ab-be80-82277155c7ee\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") "
Oct 29 23:09:13.942744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.942724    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-xks-ca\") pod \"f8341e77-16de-40ab-be80-82277155c7ee\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") "
Oct 29 23:09:13.942925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.942907    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-openshift-ca\") pod \"f8341e77-16de-40ab-be80-82277155c7ee\" (UID: \"f8341e77-16de-40ab-be80-82277155c7ee\") "
Oct 29 23:09:13.943346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.943307    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-openshift-ca" pod "f8341e77-16de-40ab-be80-82277155c7ee" (UID: "f8341e77-16de-40ab-be80-82277155c7ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:13.943778 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.943748    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.946248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.946223    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8341e77-16de-40ab-be80-82277155c7ee-kube-api-access-4z56c" pod "f8341e77-16de-40ab-be80-82277155c7ee" (UID: "f8341e77-16de-40ab-be80-82277155c7ee"). InnerVolumeSpecName "kube-api-access-4z56c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:13.946653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.946621    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-xks-ca" pod "f8341e77-16de-40ab-be80-82277155c7ee" (UID: "f8341e77-16de-40ab-be80-82277155c7ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:13.948583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:13.948563    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"]
Oct 29 23:09:14.109720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.109670    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5ld\" (UniqueName: \"kubernetes.io/projected/da18e019-4d40-4d10-923c-db375e21bf0c-kube-api-access-2s5ld\") pod \"payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:14.110039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.110016    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da18e019-4d40-4d10-923c-db375e21bf0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:14.110292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.110274    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z56c\" (UniqueName: \"kubernetes.io/projected/f8341e77-16de-40ab-be80-82277155c7ee-kube-api-access-4z56c\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:14.110418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.110406    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8341e77-16de-40ab-be80-82277155c7ee-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:14.170916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.170829    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"]
Oct 29 23:09:14.235789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.234807    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5ld\" (UniqueName: \"kubernetes.io/projected/da18e019-4d40-4d10-923c-db375e21bf0c-kube-api-access-2s5ld\") pod \"payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:14.235789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.234906    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da18e019-4d40-4d10-923c-db375e21bf0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:14.250494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.236094    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da18e019-4d40-4d10-923c-db375e21bf0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:14.270849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.267531    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5ld\" (UniqueName: \"kubernetes.io/projected/da18e019-4d40-4d10-923c-db375e21bf0c-kube-api-access-2s5ld\") pod \"payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:14.351796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.329253    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.351796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.338637    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"]
Oct 29 23:09:14.438862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.438754    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f53fb5-b929-44c4-9e79-9d1258ac3793-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.439050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.438972    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfb9\" (UniqueName: \"kubernetes.io/projected/40f53fb5-b929-44c4-9e79-9d1258ac3793-kube-api-access-bjfb9\") pod \"payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.540366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.540333    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfb9\" (UniqueName: \"kubernetes.io/projected/40f53fb5-b929-44c4-9e79-9d1258ac3793-kube-api-access-bjfb9\") pod \"payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.540570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.540550    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f53fb5-b929-44c4-9e79-9d1258ac3793-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.541261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.541241    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f53fb5-b929-44c4-9e79-9d1258ac3793-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.552161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.552134    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:14.563115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.563097    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfb9\" (UniqueName: \"kubernetes.io/projected/40f53fb5-b929-44c4-9e79-9d1258ac3793-kube-api-access-bjfb9\") pod \"payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.671474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.671432    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" event={"ID":"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c","Type":"ContainerStarted","Data":"7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90"}
Oct 29 23:09:14.671661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.671503    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" event={"ID":"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c","Type":"ContainerStarted","Data":"f5bc8db586a0486bc4e5054c56b12192b023fa62d0f1fe7048b412d5e0c3ada4"}
Oct 29 23:09:14.672896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.672849    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:14.673304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.673272    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b" event={"ID":"f8341e77-16de-40ab-be80-82277155c7ee","Type":"ContainerDied","Data":"7abc5b6702be61f0eebe748d6c1927d99e8128b4f4a349213641f86d037a9884"}
Oct 29 23:09:14.673446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.673329    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7abc5b6702be61f0eebe748d6c1927d99e8128b4f4a349213641f86d037a9884"
Oct 29 23:09:14.673446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.673356    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"
Oct 29 23:09:14.717145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.717029    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podStartSLOduration=2.717008812 podStartE2EDuration="2.717008812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:14.715847343 +0000 UTC m=+1590.338758388" watchObservedRunningTime="2026-10-29 23:09:14.717008812 +0000 UTC m=+1590.339919832"
Oct 29 23:09:14.935040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:14.935007    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"]
Oct 29 23:09:15.008014 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:15.007988    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f53fb5_b929_44c4_9e79_9d1258ac3793.slice/crio-0eb1fb94d9afa25912cb073ed52d0eb848089852364b84b644063e59f8aeb086 WatchSource:0}: Error finding container 0eb1fb94d9afa25912cb073ed52d0eb848089852364b84b644063e59f8aeb086: Status 404 returned error can't find the container with id 0eb1fb94d9afa25912cb073ed52d0eb848089852364b84b644063e59f8aeb086
Oct 29 23:09:15.009580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.009559    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"]
Oct 29 23:09:15.272597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.272519    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:15.277896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.277874    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:15.670946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.670571    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 23:09:15.683572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.683545    4827 generic.go:356] "Generic (PLEG): container finished" podID="30f142a2-daa3-4115-876d-111e248aea92" containerID="32e5f97ba36284070ba3c43217d15d7f27b35f9da4c3ca87a1aa187733dc872a" exitCode=0
Oct 29 23:09:15.683728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.683622    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" event={"ID":"30f142a2-daa3-4115-876d-111e248aea92","Type":"ContainerDied","Data":"32e5f97ba36284070ba3c43217d15d7f27b35f9da4c3ca87a1aa187733dc872a"}
Oct 29 23:09:15.683728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.683698    4827 scope.go:122] "RemoveContainer" containerID="1bf5d829141487d9ac93035a4bc9f506ea01c97dd40471f4dbd239af88ba8fbe"
Oct 29 23:09:15.685883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.685705    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" event={"ID":"40f53fb5-b929-44c4-9e79-9d1258ac3793","Type":"ContainerStarted","Data":"3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc"}
Oct 29 23:09:15.685883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.685754    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" event={"ID":"40f53fb5-b929-44c4-9e79-9d1258ac3793","Type":"ContainerStarted","Data":"0eb1fb94d9afa25912cb073ed52d0eb848089852364b84b644063e59f8aeb086"}
Oct 29 23:09:15.685883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.685823    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:15.687412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.687390    4827 generic.go:356] "Generic (PLEG): container finished" podID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerID="9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58" exitCode=0
Oct 29 23:09:15.687523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.687429    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"
Oct 29 23:09:15.687523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.687471    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" event={"ID":"7bf88017-90b5-4dd2-ab04-b1b6b9915066","Type":"ContainerDied","Data":"9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58"}
Oct 29 23:09:15.687523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.687513    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8" event={"ID":"7bf88017-90b5-4dd2-ab04-b1b6b9915066","Type":"ContainerDied","Data":"14c26a32204a72df0815567fa2bffd70d9a0fe06b94904024106e3e36489a167"}
Oct 29 23:09:15.689303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.689281    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" event={"ID":"da18e019-4d40-4d10-923c-db375e21bf0c","Type":"ContainerStarted","Data":"d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067"}
Oct 29 23:09:15.689422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.689322    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" event={"ID":"da18e019-4d40-4d10-923c-db375e21bf0c","Type":"ContainerStarted","Data":"15cf2c5e6cf8b0e8da5bbb39581d00f7de3bc63c0ef86893a94e96bf76bee6dd"}
Oct 29 23:09:15.689500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.689482    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:15.689582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.689517    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:15.690394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.690375    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:09:15.697389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.697369    4827 scope.go:122] "RemoveContainer" containerID="9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58"
Oct 29 23:09:15.707263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.707245    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 23:09:15.707806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.707760    4827 scope.go:122] "RemoveContainer" containerID="9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58"
Oct 29 23:09:15.708053 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:09:15.708030    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58\": container with ID starting with 9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58 not found: ID does not exist" containerID="9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58"
Oct 29 23:09:15.708120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.708061    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58"} err="failed to get container status \"9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58\": rpc error: code = NotFound desc = could not find container \"9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58\": container with ID starting with 9a40826911b4f11be9f262b86ada33ee809c9023a1376863cbd0d586d7f39b58 not found: ID does not exist"
Oct 29 23:09:15.723052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723028    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-data\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723090    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-certs\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723115    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-envoy\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723154    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istiod-ca-cert\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723228    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-podinfo\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723313    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-data" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.723346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723330    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-credential-socket\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723459    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-kube-api-access-twxrn\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-kube-api-access-twxrn\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723462    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-certs" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.723626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723537    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-socket\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723602    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-token\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-token\") pod \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\" (UID: \"7bf88017-90b5-4dd2-ab04-b1b6b9915066\") "
Oct 29 23:09:15.723739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723667    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istiod-ca-cert" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:15.723739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723688    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-credential-socket" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.723924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.723904    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-socket" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.725459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.725435    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.725605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.725463    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.725605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.725478    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.725605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.725492    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.725605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.725508    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.726033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.726010    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-envoy" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.726113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.726047    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-token" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:15.726113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.726099    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-kube-api-access-twxrn" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "kube-api-access-twxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:15.726223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.726206    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-podinfo" pod "7bf88017-90b5-4dd2-ab04-b1b6b9915066" (UID: "7bf88017-90b5-4dd2-ab04-b1b6b9915066"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:09:15.756731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.756669    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" podStartSLOduration=2.75665227 podStartE2EDuration="2.75665227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:15.755066692 +0000 UTC m=+1591.377977713" watchObservedRunningTime="2026-10-29 23:09:15.75665227 +0000 UTC m=+1591.379563292"
Oct 29 23:09:15.800781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.800715    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" podStartSLOduration=2.800700428 podStartE2EDuration="2.800700428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:15.796511364 +0000 UTC m=+1591.419422385" watchObservedRunningTime="2026-10-29 23:09:15.800700428 +0000 UTC m=+1591.423611478"
Oct 29 23:09:15.826223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826150    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-kube-api-access-jbqt8\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-kube-api-access-jbqt8\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826204    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-envoy\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826232    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-socket\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826297    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-istio-token\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-istio-token\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826324    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-credential-socket\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826496    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-data\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826596    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-certs\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826617    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-credential-socket" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.826673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826628    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-socket" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.826901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826741    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f142a2-daa3-4115-876d-111e248aea92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f142a2-daa3-4115-876d-111e248aea92-istiod-ca-cert\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.826901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826747    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-data" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.826901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826788    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30f142a2-daa3-4115-876d-111e248aea92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f142a2-daa3-4115-876d-111e248aea92-istio-podinfo\") pod \"30f142a2-daa3-4115-876d-111e248aea92\" (UID: \"30f142a2-daa3-4115-876d-111e248aea92\") "
Oct 29 23:09:15.827071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.826910    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-certs" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.827156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.827123    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f142a2-daa3-4115-876d-111e248aea92-istiod-ca-cert" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.827864    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.827910    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.827941    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.827964    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.827986    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f142a2-daa3-4115-876d-111e248aea92-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.828000    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxrn\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-kube-api-access-twxrn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.828010    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.828021    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf88017-90b5-4dd2-ab04-b1b6b9915066-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.828029    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.828816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.828794    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-istio-token" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:15.829031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.829005    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-envoy" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:15.829121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.829029    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-kube-api-access-jbqt8" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "kube-api-access-jbqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:15.829225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.829212    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30f142a2-daa3-4115-876d-111e248aea92-istio-podinfo" pod "30f142a2-daa3-4115-876d-111e248aea92" (UID: "30f142a2-daa3-4115-876d-111e248aea92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:09:15.931570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.931534    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.931570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.931571    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f142a2-daa3-4115-876d-111e248aea92-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.931872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.931585    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqt8\" (UniqueName: \"kubernetes.io/projected/30f142a2-daa3-4115-876d-111e248aea92-kube-api-access-jbqt8\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:15.931872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:15.931600    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f142a2-daa3-4115-876d-111e248aea92-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:16.093886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.093791    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"]
Oct 29 23:09:16.130566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.130372    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-69gc8"]
Oct 29 23:09:16.489078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.489048    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" path="/var/lib/kubelet/pods/7bf88017-90b5-4dd2-ab04-b1b6b9915066/volumes"
Oct 29 23:09:16.698506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.698480    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"
Oct 29 23:09:16.698698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.698530    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp" event={"ID":"30f142a2-daa3-4115-876d-111e248aea92","Type":"ContainerDied","Data":"cefe1b633f646988025ec0ff71575e5993acd640e8ef30988bdd36c4d814be51"}
Oct 29 23:09:16.698698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.698590    4827 scope.go:122] "RemoveContainer" containerID="32e5f97ba36284070ba3c43217d15d7f27b35f9da4c3ca87a1aa187733dc872a"
Oct 29 23:09:16.752344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.752278    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"]
Oct 29 23:09:16.763991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:16.763964    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-hrrvp"]
Oct 29 23:09:18.493518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.493490    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f142a2-daa3-4115-876d-111e248aea92" path="/var/lib/kubelet/pods/30f142a2-daa3-4115-876d-111e248aea92/volumes"
Oct 29 23:09:18.957171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.954649    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"]
Oct 29 23:09:18.958346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.957909    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy"
Oct 29 23:09:18.958346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.957988    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy"
Oct 29 23:09:18.958346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.958026    4827 cpu_manager.go:414] "containerMap: removing container" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy"
Oct 29 23:09:18.958346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.958034    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy"
Oct 29 23:09:18.958346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.958120    4827 cpu_manager.go:414] "containerMap: removing container" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy"
Oct 29 23:09:18.958346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.958128    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy"
Oct 29 23:09:18.960515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.958453    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy"
Oct 29 23:09:18.960515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.958493    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf88017-90b5-4dd2-ab04-b1b6b9915066" containerName="istio-proxy"
Oct 29 23:09:18.960515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.958510    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f142a2-daa3-4115-876d-111e248aea92" containerName="istio-proxy"
Oct 29 23:09:18.977791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.974075    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:18.980811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.979184    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"]
Oct 29 23:09:18.995398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.987441    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"]
Oct 29 23:09:18.995398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.987545    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:18.995398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:18.990754    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"]
Oct 29 23:09:19.016974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.001398    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a9ced3\""
Oct 29 23:09:19.016974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.001541    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a9ced3-dockercfg-ch57n\""
Oct 29 23:09:19.094802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.090847    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4sx\" (UniqueName: \"kubernetes.io/projected/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-kube-api-access-dp4sx\") pod \"payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.094802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.090925    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63e8edd-f701-4d64-aa15-ce42ad66743d-plugins-config-volume\") pod \"payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") " pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:19.094802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.091005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.094802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.091181    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlrg\" (UniqueName: \"kubernetes.io/projected/f63e8edd-f701-4d64-aa15-ce42ad66743d-kube-api-access-9vlrg\") pod \"payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") " pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:19.221880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.211683    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlrg\" (UniqueName: \"kubernetes.io/projected/f63e8edd-f701-4d64-aa15-ce42ad66743d-kube-api-access-9vlrg\") pod \"payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") " pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:19.225811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.224958    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4sx\" (UniqueName: \"kubernetes.io/projected/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-kube-api-access-dp4sx\") pod \"payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.225811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.225051    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63e8edd-f701-4d64-aa15-ce42ad66743d-plugins-config-volume\") pod \"payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") " pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:19.225811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.225154    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.231853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.226265    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.261899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.261866    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63e8edd-f701-4d64-aa15-ce42ad66743d-plugins-config-volume\") pod \"payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") " pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:19.313791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.313390    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlrg\" (UniqueName: \"kubernetes.io/projected/f63e8edd-f701-4d64-aa15-ce42ad66743d-kube-api-access-9vlrg\") pod \"payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") " pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:19.319464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.319235    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4sx\" (UniqueName: \"kubernetes.io/projected/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-kube-api-access-dp4sx\") pod \"payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.490711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.490627    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.610142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.603407    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:19.836147 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.836120    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"]
Oct 29 23:09:19.836742 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:19.836714    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fa0658_f0c4_47c1_ac79_4c131e96b8a4.slice/crio-a793ee7b0a8b61829cd7842fce8c5a3bbd835248c68c6bfc454d9437033adbaf WatchSource:0}: Error finding container a793ee7b0a8b61829cd7842fce8c5a3bbd835248c68c6bfc454d9437033adbaf: Status 404 returned error can't find the container with id a793ee7b0a8b61829cd7842fce8c5a3bbd835248c68c6bfc454d9437033adbaf
Oct 29 23:09:19.922034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.922007    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc" event={"ID":"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4","Type":"ContainerStarted","Data":"2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2"}
Oct 29 23:09:19.922142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.922069    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc" event={"ID":"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4","Type":"ContainerStarted","Data":"a793ee7b0a8b61829cd7842fce8c5a3bbd835248c68c6bfc454d9437033adbaf"}
Oct 29 23:09:19.922142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.922117    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:19.969703 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:19.969665    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63e8edd_f701_4d64_aa15_ce42ad66743d.slice/crio-7d4434525524edb8431f1095ae834854833cce1e56fce05e69354fc607542aef WatchSource:0}: Error finding container 7d4434525524edb8431f1095ae834854833cce1e56fce05e69354fc607542aef: Status 404 returned error can't find the container with id 7d4434525524edb8431f1095ae834854833cce1e56fce05e69354fc607542aef
Oct 29 23:09:19.970542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.970488    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"]
Oct 29 23:09:19.971732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:19.971681    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc" podStartSLOduration=1.9716649689999999 podStartE2EDuration="1.971664969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:19.966913061 +0000 UTC m=+1595.589824080" watchObservedRunningTime="2026-10-29 23:09:19.971664969 +0000 UTC m=+1595.594575982"
Oct 29 23:09:20.932173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:20.932131    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn" event={"ID":"f63e8edd-f701-4d64-aa15-ce42ad66743d","Type":"ContainerStarted","Data":"abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b"}
Oct 29 23:09:20.932650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:20.932195    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn" event={"ID":"f63e8edd-f701-4d64-aa15-ce42ad66743d","Type":"ContainerStarted","Data":"7d4434525524edb8431f1095ae834854833cce1e56fce05e69354fc607542aef"}
Oct 29 23:09:20.932650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:20.932225    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:20.973141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:20.973099    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn" podStartSLOduration=2.973087311 podStartE2EDuration="2.973087311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:20.972097256 +0000 UTC m=+1596.595008288" watchObservedRunningTime="2026-10-29 23:09:20.973087311 +0000 UTC m=+1596.595998330"
Oct 29 23:09:21.207397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.207321    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"]
Oct 29 23:09:21.211657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.211626    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.227992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.227965    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-af237d-serving-cert\""
Oct 29 23:09:21.228100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.228048    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:09:21.228146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.228095    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xrwtg\""
Oct 29 23:09:21.233263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.233239    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"]
Oct 29 23:09:21.269554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.269518    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9534671-b20f-432d-a550-bcf1d18bc280-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.269699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.269650    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlqm\" (UniqueName: \"kubernetes.io/projected/b9534671-b20f-432d-a550-bcf1d18bc280-kube-api-access-7vlqm\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.269825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.269805    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.269871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.269848    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.371636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.371603    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9534671-b20f-432d-a550-bcf1d18bc280-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.371844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.371742    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlqm\" (UniqueName: \"kubernetes.io/projected/b9534671-b20f-432d-a550-bcf1d18bc280-kube-api-access-7vlqm\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.371891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.371854    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.371891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.371879    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.372284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.372261    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9534671-b20f-432d-a550-bcf1d18bc280-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.374334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.374308    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.374444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.374344    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.395860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.395832    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlqm\" (UniqueName: \"kubernetes.io/projected/b9534671-b20f-432d-a550-bcf1d18bc280-kube-api-access-7vlqm\") pod \"maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.535414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.526072    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:21.890365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.890338    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"]
Oct 29 23:09:21.891056 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:21.891034    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9534671_b20f_432d_a550_bcf1d18bc280.slice/crio-4ffe81fbbd66954a24eba10e3e311e9ee8fff47dc54ef7053a1e68819a0de3bf WatchSource:0}: Error finding container 4ffe81fbbd66954a24eba10e3e311e9ee8fff47dc54ef7053a1e68819a0de3bf: Status 404 returned error can't find the container with id 4ffe81fbbd66954a24eba10e3e311e9ee8fff47dc54ef7053a1e68819a0de3bf
Oct 29 23:09:21.892548 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.892531    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:09:21.940259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.940227    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp" event={"ID":"b9534671-b20f-432d-a550-bcf1d18bc280","Type":"ContainerStarted","Data":"4ffe81fbbd66954a24eba10e3e311e9ee8fff47dc54ef7053a1e68819a0de3bf"}
Oct 29 23:09:21.961756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.961721    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"]
Oct 29 23:09:21.968921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.968898    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.001793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:21.996394    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b0c3c3-openshift-default-dockercfg-nzrxs\""
Oct 29 23:09:22.008528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.008496    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"]
Oct 29 23:09:22.085414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.085371    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d06c4719-725f-4a22-8359-8d5d2de03a60-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.085606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.085475    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.085606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.085572    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.085723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.085610    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.085723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.085646    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.085908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.085888    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.086013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.085997    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ncvz\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-kube-api-access-2ncvz\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.086080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.086055    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.086264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.086232    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.098267 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.098230    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"]
Oct 29 23:09:22.098482 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.098445    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" containerID="cri-o://e6ed3b76546b25e70b6ec67ca0085f3b2cdc1144545451fb87c3b6e7b4a55713" gracePeriod=30
Oct 29 23:09:22.188445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188360    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188454    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ncvz\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-kube-api-access-2ncvz\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188503    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188538    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188571    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d06c4719-725f-4a22-8359-8d5d2de03a60-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188659    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188705    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188723    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188761    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.188851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.188820    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.189120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.189098    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.189171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.189141    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.189293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.189264    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.189397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.189379    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d06c4719-725f-4a22-8359-8d5d2de03a60-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.190973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.190952    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.191107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.191090    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.213323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.213264    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ncvz\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-kube-api-access-2ncvz\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.215112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.214984    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.321205 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.321120    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:22.695243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.691660    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:09:22.695243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.694187    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:09:22.697911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.697310    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:09:22.698508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.698169    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:09:22.949484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.949404    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp" event={"ID":"b9534671-b20f-432d-a550-bcf1d18bc280","Type":"ContainerStarted","Data":"1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71"}
Oct 29 23:09:22.949923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:22.949484    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:23.063115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.047073    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp" podStartSLOduration=1.289295943 podStartE2EDuration="2.047050019s" totalImagesPullingTime="757.754076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:23.011561917 +0000 UTC m=+1598.634472937" watchObservedRunningTime="2026-10-29 23:09:23.047050019 +0000 UTC m=+1598.669961041"
Oct 29 23:09:23.084088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.084058    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"]
Oct 29 23:09:23.088174 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:23.088147    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06c4719_725f_4a22_8359_8d5d2de03a60.slice/crio-0fe74246e5af915168883461b37b8292c30ff8e70740d418e17322479488560d WatchSource:0}: Error finding container 0fe74246e5af915168883461b37b8292c30ff8e70740d418e17322479488560d: Status 404 returned error can't find the container with id 0fe74246e5af915168883461b37b8292c30ff8e70740d418e17322479488560d
Oct 29 23:09:23.091043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.091008    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:23.091139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.091091    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:23.091139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.091122    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:23.126841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.121728    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"]
Oct 29 23:09:23.154801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.149889    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.187502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.166199    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-4504f1-serving-cert\""
Oct 29 23:09:23.212991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.208152    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"]
Oct 29 23:09:23.348918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.348880    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c64dd95b-d357-488e-9aa4-8d3376e2cba3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.349104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.348957    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmqb5\" (UniqueName: \"kubernetes.io/projected/c64dd95b-d357-488e-9aa4-8d3376e2cba3-kube-api-access-fmqb5\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.349104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.349047    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.349221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.349169    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.453825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.451431    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.453825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.451630    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c64dd95b-d357-488e-9aa4-8d3376e2cba3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.453825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.451702    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqb5\" (UniqueName: \"kubernetes.io/projected/c64dd95b-d357-488e-9aa4-8d3376e2cba3-kube-api-access-fmqb5\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.453825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.451815    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.461813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.459753    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c64dd95b-d357-488e-9aa4-8d3376e2cba3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.469886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.464720    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.498795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.492147    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.513790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.505491    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqb5\" (UniqueName: \"kubernetes.io/projected/c64dd95b-d357-488e-9aa4-8d3376e2cba3-kube-api-access-fmqb5\") pod \"maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.513790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.509034    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:23.639085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.638888    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"]
Oct 29 23:09:23.645332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.645305    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.653505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.653484    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"]
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.655603    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.655663    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.655822    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e841572-ee8f-416c-aa8f-2eeeb348423e-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.655894    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcdj\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-kube-api-access-6wcdj\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.655925    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.655961    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.656005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.656028    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.657700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.656052    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.659095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.659073    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"]
Oct 29 23:09:23.659249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.659232    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.680091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.680057    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"]
Oct 29 23:09:23.756939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.756855    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddsx\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-kube-api-access-kddsx\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.756939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.756895    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.757176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757023    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e841572-ee8f-416c-aa8f-2eeeb348423e-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757073    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.757176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757140    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.757329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757192    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcdj\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-kube-api-access-6wcdj\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757253    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757312    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757357    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.757483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757391    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757425    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757459    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757551    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.757674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757608    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757697    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.757808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757728    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757779    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757814    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e841572-ee8f-416c-aa8f-2eeeb348423e-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757833    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757845    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.757985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757862    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.757985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.757934    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.758253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.758028    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.760101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.760073    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.760408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.760393    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.780342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.780312    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.780504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.780395    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcdj\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-kube-api-access-6wcdj\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:23.859488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.859456    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.859603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.859516    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.859603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.859567    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.859603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.859600    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddsx\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-kube-api-access-kddsx\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.859834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.859808    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.859960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.859940    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.859992    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.860020    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.860071    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.860115    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.860225    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.860278    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.860305    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.860574 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:23.860548    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64dd95b_d357_488e_9aa4_8d3376e2cba3.slice/crio-222c95bd39d43ac0cd5fc6106076c4d2e9762a9f8c4c5e522a2e8c04cba0a726 WatchSource:0}: Error finding container 222c95bd39d43ac0cd5fc6106076c4d2e9762a9f8c4c5e522a2e8c04cba0a726: Status 404 returned error can't find the container with id 222c95bd39d43ac0cd5fc6106076c4d2e9762a9f8c4c5e522a2e8c04cba0a726
Oct 29 23:09:23.860687 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.860586    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.861057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.861035    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"]
Oct 29 23:09:23.862538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.862512    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.862634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.862527    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.882652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.882618    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.882976 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.882956    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddsx\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-kube-api-access-kddsx\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:23.959101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.959060    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf" event={"ID":"c64dd95b-d357-488e-9aa4-8d3376e2cba3","Type":"ContainerStarted","Data":"222c95bd39d43ac0cd5fc6106076c4d2e9762a9f8c4c5e522a2e8c04cba0a726"}
Oct 29 23:09:23.961034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.961003    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" event={"ID":"d06c4719-725f-4a22-8359-8d5d2de03a60","Type":"ContainerStarted","Data":"e3a2070c02710eca274382963ceff2c14f999130a263c9d8b9f21ec14e8a7a81"}
Oct 29 23:09:23.961184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.961079    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" event={"ID":"d06c4719-725f-4a22-8359-8d5d2de03a60","Type":"ContainerStarted","Data":"0fe74246e5af915168883461b37b8292c30ff8e70740d418e17322479488560d"}
Oct 29 23:09:23.984107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:23.984068    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:24.033458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.031618    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:24.322981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.321930    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:24.379660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.379614    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:24.434708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.434646    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podStartSLOduration=3.434623121 podStartE2EDuration="3.434623121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:24.030110212 +0000 UTC m=+1599.653021234" watchObservedRunningTime="2026-10-29 23:09:24.434623121 +0000 UTC m=+1600.057534141"
Oct 29 23:09:24.570850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.566145    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:24.574789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.572177    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:24.574789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.572249    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:24.619847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.619814    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"]
Oct 29 23:09:24.692474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.674218    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:24.692474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.674338    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:24.692474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.674405    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:24.797728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:24.797693    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"]
Oct 29 23:09:25.064954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.064913    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf" event={"ID":"c64dd95b-d357-488e-9aa4-8d3376e2cba3","Type":"ContainerStarted","Data":"c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb"}
Oct 29 23:09:25.072369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.072313    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:25.201617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.201563    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" event={"ID":"b5aca5ed-64a2-4f72-9d40-02130ffc176e","Type":"ContainerStarted","Data":"67022a2ba9432f194df222f94195142ce7e9606891cc5d0d23c8b7d05e5d2614"}
Oct 29 23:09:25.244791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.219921    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" event={"ID":"3e841572-ee8f-416c-aa8f-2eeeb348423e","Type":"ContainerStarted","Data":"7d202e04dff174a8300509fb9980bad55387d2418ad2eed269180a6dd8b281d4"}
Oct 29 23:09:25.244791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.219990    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:25.244791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.229481    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:09:25.392808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.368805    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf" podStartSLOduration=2.009718624 podStartE2EDuration="2.36878293s" totalImagesPullingTime="359.064306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:25.137657901 +0000 UTC m=+1600.760568989" watchObservedRunningTime="2026-10-29 23:09:25.36878293 +0000 UTC m=+1600.991693951"
Oct 29 23:09:25.427812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.395935    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"]
Oct 29 23:09:25.612865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.609395    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:25.655285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.655239    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"]
Oct 29 23:09:25.666104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.666071    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.768977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.768940    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72v4w\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-kube-api-access-72v4w\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.769283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.769262    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.769486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.769466    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f528dc4-4901-428a-8f32-591204781c07-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.769638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.769623    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f528dc4-4901-428a-8f32-591204781c07-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.769798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.769781    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.770001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.769985    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.770117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.770102    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.770248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.770234    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.777883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.777843    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.807023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.800742    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.880995    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881046    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881107    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881158    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881257    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72v4w\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-kube-api-access-72v4w\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881352    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881453    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f528dc4-4901-428a-8f32-591204781c07-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881516    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f528dc4-4901-428a-8f32-591204781c07-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.881545    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.917974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.882535    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:25.936815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.924997    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"]
Oct 29 23:09:25.936815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.925051    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"]
Oct 29 23:09:25.936815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.925160    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:25.982412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.954023    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.052134    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f528dc4-4901-428a-8f32-591204781c07-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.962358    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.024926    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064128    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsssd\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-kube-api-access-lsssd\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064215    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064320    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064359    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064398    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8972826-8c7b-466f-9b19-818336fde77c-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064430    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064502    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8972826-8c7b-466f-9b19-818336fde77c-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.064791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.064556    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.128825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.091340    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f528dc4-4901-428a-8f32-591204781c07-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.128825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:25.961700    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.128825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.125581    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72v4w\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-kube-api-access-72v4w\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.195848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.171614    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.196112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.176634    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.204794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.196349    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsssd\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-kube-api-access-lsssd\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.205124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.205095    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.212960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.212930    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.216793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.213127    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.265219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.265182    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8972826-8c7b-466f-9b19-818336fde77c-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.265478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.265451    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.265700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.265680    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8972826-8c7b-466f-9b19-818336fde77c-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.294789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.269864    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.299789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.177126    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.329790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.304544    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8972826-8c7b-466f-9b19-818336fde77c-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.355789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.337195    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.380791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.363302    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.380791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.363581    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.380791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.365028    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.380791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.365333    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.380791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.318694    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsssd\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-kube-api-access-lsssd\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.458847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.404558    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.512790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.509668    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8972826-8c7b-466f-9b19-818336fde77c-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.688079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.688030    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:26.690849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.690822    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:26.721197 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.698342    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:09:26.721197 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.698406    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:26.723469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.723439    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:26.849450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.790087    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:26.852965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:26.813481    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:09:27.070711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.070600    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:27.071852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.071819    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:09:27.132793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.072883    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:27.133422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.072910    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:09:27.164791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.118939    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:27.231512 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.119010    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:09:27.232097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.119040    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:09:27.232576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.119059    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:09:27.731789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.716923    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:27.834646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:27.819271    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.008629    4827 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.008710    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.033260    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.046511    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.046572    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:09:28.060059    4827 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.569s"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060095    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" event={"ID":"3e841572-ee8f-416c-aa8f-2eeeb348423e","Type":"ContainerStarted","Data":"29d83d68183830519d0dde6beee5be7dd9624bc04c91a2d3350fa23a740ffafc"}
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060188    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060248    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060293    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060328    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060364    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060406    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060431    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060457    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.060489    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:09:28.098660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.079920    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:28.110968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.110919    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:09:28.255904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.235268    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:28.255904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.252357    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:09:28.255904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.252717    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:28.348134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.294089    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:28.348134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.304320    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:09:28.551787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.541207    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podStartSLOduration=5.541187192 podStartE2EDuration="5.541187192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:28.006353487 +0000 UTC m=+1603.629264532" watchObservedRunningTime="2026-10-29 23:09:28.541187192 +0000 UTC m=+1604.164098216"
Oct 29 23:09:28.596792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.583377    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:09:28.664900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.599886    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:28.698217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.691956    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:09:28.698217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.692004    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:09:28.698217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.692038    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:09:28.698217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.692054    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:09:28.698217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.692082    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:09:28.698217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.692108    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:09:28.698217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.692135    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:09:28.876792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.862942    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 23:09:28.876792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.863002    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 23:09:28.941210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.937463    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" event={"ID":"b5aca5ed-64a2-4f72-9d40-02130ffc176e","Type":"ContainerStarted","Data":"9def025358a96bc7f6fd785148ae8b1c7c248db4561821c59815875eca0dde3f"}
Oct 29 23:09:28.950710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.944045    4827 generic.go:356] "Generic (PLEG): container finished" podID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerID="e6ed3b76546b25e70b6ec67ca0085f3b2cdc1144545451fb87c3b6e7b4a55713" exitCode=0
Oct 29 23:09:28.950710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:28.945151    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" event={"ID":"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e","Type":"ContainerDied","Data":"e6ed3b76546b25e70b6ec67ca0085f3b2cdc1144545451fb87c3b6e7b4a55713"}
Oct 29 23:09:29.042861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.041325    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:29.042861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.041381    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:29.050873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.050812    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:29.052862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.052829    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:29.053028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.053009    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:29.053481 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.053456    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:29.053656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.053634    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:29.053804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.051173    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:09:29.054062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.054045    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:29.058169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.058146    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"]
Oct 29 23:09:29.408801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.393304    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podStartSLOduration=6.393283838 podStartE2EDuration="6.393283838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:29.354337404 +0000 UTC m=+1604.977248450" watchObservedRunningTime="2026-10-29 23:09:29.393283838 +0000 UTC m=+1605.016194862"
Oct 29 23:09:29.545067 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.545037    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 23:09:29.607817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.603338    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-podinfo\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.607817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.603447    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.607817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.603513    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.607817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.603561    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-kube-api-access-jdwn9\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-kube-api-access-jdwn9\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.607817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.603651    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-credential-socket\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.607817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.603680    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-certs\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.733960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.686304    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-data\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.733960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.686569    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-envoy\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.733960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.686635    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-token\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-token\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.733960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.721920    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-certs" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:29.794952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.794831    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"]
Oct 29 23:09:29.824171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.806603    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:29.824171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.810835    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.826257    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.826339    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\") pod \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\" (UID: \"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e\") "
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.827475    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:29.827572    4827 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e/volumes/kubernetes.io~configmap/istiod-ca-cert
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.827585    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:29.827638    4827 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e/volumes/kubernetes.io~empty-dir/workload-socket
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.827649    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.836872    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-data" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:29.844114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.843128    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-credential-socket" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:29.932527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.913237    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:29.932527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.913338    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:29.932527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.913380    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:29.957578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.957540    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.957922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.957898    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.958032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.958019    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.958157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.958145    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.998838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.990702    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-podinfo" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:09:29.998838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.991245    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:29.998838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:29.991293    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:30.056527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.056491    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:30.056917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.056876    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:30.059984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.059958    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-kube-api-access-jdwn9" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "kube-api-access-jdwn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:30.082022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.081982    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-envoy" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:30.083421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.083378    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-token" pod "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" (UID: "aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:30.120377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.091202    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwn9\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-kube-api-access-jdwn9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:30.123039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.123013    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:30.123327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.123310    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:30.123504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.123489    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:09:30.244073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.244029    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" event={"ID":"6f528dc4-4901-428a-8f32-591204781c07","Type":"ContainerStarted","Data":"dd1e16b3401b8f6d3fc3f43fd69194ccfa34622beffed7a40593bc82412b1e2f"}
Oct 29 23:09:30.244275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.244107    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" event={"ID":"6f528dc4-4901-428a-8f32-591204781c07","Type":"ContainerStarted","Data":"d21c5cfc30a080715ab2e62be95e5a126a15064dab84054a5560d02d658ae5a2"}
Oct 29 23:09:30.322938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.322756    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" event={"ID":"a8972826-8c7b-466f-9b19-818336fde77c","Type":"ContainerStarted","Data":"0f8b953fc5d5813bd5b8899510cb3cad3ec68e59d89511c2c6704d5a77f68b00"}
Oct 29 23:09:30.326059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.326026    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84" event={"ID":"aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e","Type":"ContainerDied","Data":"fc6704373ddfcc59f1e41e8ed896105331de9432c932e59160ec29c90ab895c1"}
Oct 29 23:09:30.326223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.326093    4827 scope.go:122] "RemoveContainer" containerID="e6ed3b76546b25e70b6ec67ca0085f3b2cdc1144545451fb87c3b6e7b4a55713"
Oct 29 23:09:30.328346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.328311    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"
Oct 29 23:09:30.335826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.335512    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podStartSLOduration=5.335493917 podStartE2EDuration="5.335493917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:30.32640262 +0000 UTC m=+1605.949313643" watchObservedRunningTime="2026-10-29 23:09:30.335493917 +0000 UTC m=+1605.958404936"
Oct 29 23:09:30.588906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.588816    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"]
Oct 29 23:09:30.589465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.589447    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lcd84"]
Oct 29 23:09:30.757822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.757792    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:30.758679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.758654    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:30.758888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.758868    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:30.938957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.937755    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:09:30.988801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.988258    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:30.988801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:30.988316    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:31.053510 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.053466    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:31.054080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.053876    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:31.493791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.490371    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" event={"ID":"a8972826-8c7b-466f-9b19-818336fde77c","Type":"ContainerStarted","Data":"471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64"}
Oct 29 23:09:31.748881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.741785    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:31.748881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.741947    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:31.748881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.742006    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:31.778798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.773083    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:31.778798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.773162    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:31.957308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.955927    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:09:31.991383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.991350    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:31.991700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:31.991670    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:32.016807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.016680    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podStartSLOduration=7.016660021 podStartE2EDuration="7.016660021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:31.607128428 +0000 UTC m=+1607.230039471" watchObservedRunningTime="2026-10-29 23:09:32.016660021 +0000 UTC m=+1607.639571044"
Oct 29 23:09:32.039190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.039149    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:32.039449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.039424    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:32.809792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.794921    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:32.809792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.795003    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:32.834347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.807934    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:32.900793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.900046    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:32.900793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.900116    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:32.999298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.998150    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:32.999298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:32.999259    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:33.015817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.003214    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:09:33.082873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.081964    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:33.082873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.082033    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:33.156787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.153024    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:09:33.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.203837    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:33.247790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.242798    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:33.272872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.242731    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" path="/var/lib/kubelet/pods/aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e/volumes"
Oct 29 23:09:33.273814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.268136    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 29 23:09:33.382705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.382669    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:09:33.455859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.455827    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:09:33.793849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.790136    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:33.816429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.816389    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:34.011792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.010872    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:34.011792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.817572    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.011792 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:09:33.817421    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:09:34.011792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.907656    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:09:34.011792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.817682    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:34.011792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.011544    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:34.011792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.004162    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:09:34.012351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.905867    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.012351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.916932    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.012468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.012448    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:09:34.012529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.012509    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:09:34.016882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.004106    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.017405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:33.988185    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:34.020787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.017561    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:34.048917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.020067    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:09:34.121903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.004189    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.123567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.056127    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.131789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.127877    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:09:34.150830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.128711    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:34.151135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.151103    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:34.204792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.154669    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.204792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.155835    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:09:34.235790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.213235    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.274791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.269204    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"]
Oct 29 23:09:34.274791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.270319    4827 cpu_manager.go:414] "containerMap: removing container" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy"
Oct 29 23:09:34.274791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.270341    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy"
Oct 29 23:09:34.398580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.385422    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:34.398955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.398927    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacfb504-ddb6-4ec7-a13d-cc6488c2cc9e" containerName="istio-proxy"
Oct 29 23:09:34.760819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.711338    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:34.961306 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:34.880974    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:35.059061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.036868    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:09:35.059061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.037171    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:35.083640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.077016    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:09:35.084291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.079911    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:09:35.084601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.079944    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:09:35.127630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.127567    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:35.132029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.131977    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:35.135648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.133619    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"]
Oct 29 23:09:35.186475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.186444    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-32717a-serving-cert\""
Oct 29 23:09:35.214800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.196395    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:35.215476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.215435    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:35.238798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.217642    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:35.352363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.352170    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:35.352600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.232994    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:35.353358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.351946    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfrt\" (UniqueName: \"kubernetes.io/projected/ae64c2b6-7120-4267-86bd-b3d8e6140252-kube-api-access-nhfrt\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.353620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.353579    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae64c2b6-7120-4267-86bd-b3d8e6140252-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.353880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.353859    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.354178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.354156    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.354693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.222282    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:35.355443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.231972    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:35.355608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.355584    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:35.355730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.234332    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:35.355857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.355838    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:35.388169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.388046    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:35.438380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.438349    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:09:35.438688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.438674    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:09:35.458011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.439030    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:09:35.458011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.439091    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:09:35.458011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.439126    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:09:35.458011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.439165    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:09:35.458011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.439196    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:09:35.461788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.461746    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfrt\" (UniqueName: \"kubernetes.io/projected/ae64c2b6-7120-4267-86bd-b3d8e6140252-kube-api-access-nhfrt\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.462045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.462024    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae64c2b6-7120-4267-86bd-b3d8e6140252-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.462247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.462233    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.469802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.462455    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.471488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.471456    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae64c2b6-7120-4267-86bd-b3d8e6140252-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.619607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.610524    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfrt\" (UniqueName: \"kubernetes.io/projected/ae64c2b6-7120-4267-86bd-b3d8e6140252-kube-api-access-nhfrt\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.670787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.656460    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.793798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.793132    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:35.793798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.793189    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:35.800811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.794696    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:35.800811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.794740    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:35.837298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.837266    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:35.870916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:35.870708    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:09:36.047338 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.047304    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:36.047891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.047857    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.091243    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.091303    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.118744    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.135077    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.135180    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.165145    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.194855    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 23:09:36.202065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.196031    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:09:36.218830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.206855    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:09:36.501744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.501633    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:09:36.503885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.503833    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:36.505460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.505432    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:09:36.818865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.808568    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:36.818865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.808642    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:36.818865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.809003    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:36.818865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.809067    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:36.873789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.873730    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:09:36.908349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.897512    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:09:36.908699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.908680    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:09:36.908906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:36.908889    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:09:37.090139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.090107    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:09:37.096895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.096865    4827 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 29 23:09:37.097118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.097091    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 29 23:09:37.135892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.135335    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:37.160099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.159176    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:09:37.190791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.181421    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:37.202423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.200909    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:37.341506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.341016    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:37.352860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.352399    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:09:37.608822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.608751    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:37.845854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.643874    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:37.845854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.720132    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:09:37.845854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.780385    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:37.845854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.780436    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:37.845854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.813147    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:37.845854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.814800    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:37.847209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.847135    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:09:37.859152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.859118    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:37.859437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.859404    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:37.908140 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.908094    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:37.958428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:37.948916    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:09:38.079635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.079599    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"]
Oct 29 23:09:38.175384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.169630    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:09:38.227660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.201531    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:38.250678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.250650    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:09:38.253078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.253055    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:09:38.560861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.559850    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:38.560861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.560491    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:38.561420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.561396    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:09:38.740849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.740816    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:38.740980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.740884    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:38.760184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.760148    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:09:38.760358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.760204    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:09:38.989814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.988150    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl" event={"ID":"ae64c2b6-7120-4267-86bd-b3d8e6140252","Type":"ContainerStarted","Data":"82833d08df761b27bd7d9bc62fce776c8d6d453e2a279bf52ef4f751bc095cb1"}
Oct 29 23:09:38.989814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.988223    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl" event={"ID":"ae64c2b6-7120-4267-86bd-b3d8e6140252","Type":"ContainerStarted","Data":"0a88acab14165bdf9968a30a2f65eb89cae4fbb8a6db4a529559c4ef9b6f85d2"}
Oct 29 23:09:38.989814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:38.989321    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:39.034892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:39.034825    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl" podStartSLOduration=5.63678632 podStartE2EDuration="6.034804479s" totalImagesPullingTime="398.018159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:39.030552076 +0000 UTC m=+1614.653463125" watchObservedRunningTime="2026-10-29 23:09:39.034804479 +0000 UTC m=+1614.657715500"
Oct 29 23:09:39.740021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:39.739980    4827 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 29 23:09:39.740440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:39.740048    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 29 23:09:39.899637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:39.899595    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:39.900088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:39.900057    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:39.900797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:39.900746    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:09:40.744760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:40.744729    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:40.745255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:40.745241    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:40.746025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:40.746004    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:09:41.977380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:41.977350    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"]
Oct 29 23:09:41.981881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:41.981865    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:41.997720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:41.997698    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-a9ced3-serving-cert\""
Oct 29 23:09:42.016350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.016320    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"]
Oct 29 23:09:42.039505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.039478    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-openshift-service-ca\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.039683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.039584    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-maas-api-tls\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.039821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.039748    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-metrics-certs\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.040019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.039993    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljd9n\" (UniqueName: \"kubernetes.io/projected/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-kube-api-access-ljd9n\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.141808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.141749    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-openshift-service-ca\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.141966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.141860    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-maas-api-tls\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.142036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.141973    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-metrics-certs\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.142036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.141999    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljd9n\" (UniqueName: \"kubernetes.io/projected/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-kube-api-access-ljd9n\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.142727 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.142698    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-openshift-service-ca\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.144895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.144872    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-metrics-certs\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.145001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.144893    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-maas-api-tls\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.193792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.177288    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljd9n\" (UniqueName: \"kubernetes.io/projected/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-kube-api-access-ljd9n\") pod \"maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.317896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.299059    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:42.690602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:42.690571    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"]
Oct 29 23:09:42.691523 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:42.691494    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59d5004_e05b_495e_ad82_fb7d0c5dfa66.slice/crio-b7ee6fcc122239cf7af9eb31b46a4116b3fd5b9957ce5e4cb057a586fb605436 WatchSource:0}: Error finding container b7ee6fcc122239cf7af9eb31b46a4116b3fd5b9957ce5e4cb057a586fb605436: Status 404 returned error can't find the container with id b7ee6fcc122239cf7af9eb31b46a4116b3fd5b9957ce5e4cb057a586fb605436
Oct 29 23:09:43.024158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:43.024068    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr" event={"ID":"d59d5004-e05b-495e-ad82-fb7d0c5dfa66","Type":"ContainerStarted","Data":"b7ee6fcc122239cf7af9eb31b46a4116b3fd5b9957ce5e4cb057a586fb605436"}
Oct 29 23:09:44.053477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:44.052357    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr" event={"ID":"d59d5004-e05b-495e-ad82-fb7d0c5dfa66","Type":"ContainerStarted","Data":"b79d570e24082d1591a994b47a81c0a27583e27e77327871a7865e80f6b04372"}
Oct 29 23:09:44.053477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:44.053349    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:44.095998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:44.095946    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr" podStartSLOduration=2.6795566490000002 podStartE2EDuration="3.095931904s" totalImagesPullingTime="416.375255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:44.094352275 +0000 UTC m=+1619.717263296" watchObservedRunningTime="2026-10-29 23:09:44.095931904 +0000 UTC m=+1619.718842924"
Oct 29 23:09:46.010650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:46.010620    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:09:50.065439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:50.065411    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:09:52.204027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.203986    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"]
Oct 29 23:09:52.211887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.211857    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:52.236635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.236608    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"]
Oct 29 23:09:52.242237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.242197    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.252449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.252408    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"]
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.408955    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a092b8-5a06-4af1-9db7-870948480836-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409136    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409257    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409307    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409378    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409411    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409488    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409567    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltv8v\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-kube-api-access-ltv8v\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:52.410791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:52.409682    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a092b8-5a06-4af1-9db7-870948480836-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.304263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.304222    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a092b8-5a06-4af1-9db7-870948480836-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.349789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.339625    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633e2112-6693-4fdb-980a-8b47b2cd7898-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.388047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.325175    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a092b8-5a06-4af1-9db7-870948480836-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.388312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.377153    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:53.404018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.403978    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.404748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.404723    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz98k\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-kube-api-access-hz98k\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.416194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.416148    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a092b8-5a06-4af1-9db7-870948480836-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.416542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.416517    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.416738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.416723    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.417035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.417005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.420786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.417327    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.456109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.456067    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.456502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.456469    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.456713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.456696    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.456917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.456899    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.457071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.457048    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.457309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.457278    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.462667 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.462643    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltv8v\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-kube-api-access-ltv8v\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.462879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.459004    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.464323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.464273    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.469060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.469002    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.469252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.469082    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.469252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.469171    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.487992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.487943    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.508846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.508816    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.524566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.514169    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a092b8-5a06-4af1-9db7-870948480836-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.524566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.516310    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltv8v\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-kube-api-access-ltv8v\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.524566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.519152    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"]
Oct 29 23:09:53.526523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.526492    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:53.570610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570523    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.570827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570608    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.570827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570736    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633e2112-6693-4fdb-980a-8b47b2cd7898-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.570827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570794    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570827    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz98k\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-kube-api-access-hz98k\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570895    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570929    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.570983    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.571020    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.571047    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.571205    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.571429    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571548 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.571530    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633e2112-6693-4fdb-980a-8b47b2cd7898-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.571664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.571644    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.573421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.573395    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.574139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.574116    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.599965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.599937    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.600176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.600153    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz98k\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-kube-api-access-hz98k\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.737818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.728883    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:53.815397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:53.815363    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:54.106304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.106277    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"]
Oct 29 23:09:54.107754 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:54.107731    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633e2112_6693_4fdb_980a_8b47b2cd7898.slice/crio-5745e8feb04a1607d98b4badbec4aed6d5ca115ba8e588a51a602d3089b34290 WatchSource:0}: Error finding container 5745e8feb04a1607d98b4badbec4aed6d5ca115ba8e588a51a602d3089b34290: Status 404 returned error can't find the container with id 5745e8feb04a1607d98b4badbec4aed6d5ca115ba8e588a51a602d3089b34290
Oct 29 23:09:54.110857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.110821    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:54.110948 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.110899    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:54.110948 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.110930    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:54.185026 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:09:54.184999    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a092b8_5a06_4af1_9db7_870948480836.slice/crio-dae00e199aaae9846dab0958a7330ccf64dc1b3fdc3358f350640b2d7322d5f8 WatchSource:0}: Error finding container dae00e199aaae9846dab0958a7330ccf64dc1b3fdc3358f350640b2d7322d5f8: Status 404 returned error can't find the container with id dae00e199aaae9846dab0958a7330ccf64dc1b3fdc3358f350640b2d7322d5f8
Oct 29 23:09:54.185252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.185097    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"]
Oct 29 23:09:54.187953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.187919    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:54.188056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.187993    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:54.188056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.188025    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:09:54.577122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.577089    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" event={"ID":"b8a092b8-5a06-4af1-9db7-870948480836","Type":"ContainerStarted","Data":"34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce"}
Oct 29 23:09:54.577516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.577143    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" event={"ID":"b8a092b8-5a06-4af1-9db7-870948480836","Type":"ContainerStarted","Data":"dae00e199aaae9846dab0958a7330ccf64dc1b3fdc3358f350640b2d7322d5f8"}
Oct 29 23:09:54.577516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.577168    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" event={"ID":"633e2112-6693-4fdb-980a-8b47b2cd7898","Type":"ContainerStarted","Data":"e97f672fd604ff3bfb1e0ad501d13fe417e11679a6c8e4e3329480561ace7b08"}
Oct 29 23:09:54.577516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.577186    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" event={"ID":"633e2112-6693-4fdb-980a-8b47b2cd7898","Type":"ContainerStarted","Data":"5745e8feb04a1607d98b4badbec4aed6d5ca115ba8e588a51a602d3089b34290"}
Oct 29 23:09:54.603967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.603907    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podStartSLOduration=2.603890756 podStartE2EDuration="2.603890756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:54.599935178 +0000 UTC m=+1630.222846199" watchObservedRunningTime="2026-10-29 23:09:54.603890756 +0000 UTC m=+1630.226801778"
Oct 29 23:09:54.649430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.648994    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podStartSLOduration=2.6489752810000002 podStartE2EDuration="2.648975281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:54.644644248 +0000 UTC m=+1630.267555269" watchObservedRunningTime="2026-10-29 23:09:54.648975281 +0000 UTC m=+1630.271886297"
Oct 29 23:09:54.729870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.729840    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:09:54.815867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:54.815841    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:09:55.736311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:55.736242    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:09:55.736951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:55.736923    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:09:55.818339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:55.818300    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:09:55.818522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:55.818366    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:09:56.730793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:56.730273    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:09:56.730793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:56.730337    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:09:56.819288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:56.819234    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:09:56.819855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:56.819303    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:09:57.730149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:57.730100    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:09:57.730371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:57.730168    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:09:57.817793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:57.816202    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:09:57.817793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:57.816265    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:09:58.730126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:58.730092    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:09:58.730565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:58.730163    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:09:59.162883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:59.145900    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:09:59.162883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:59.145963    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:09:59.805516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:59.805482    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:09:59.821936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:59.821896    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:09:59.822511 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:59.816104    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:09:59.822707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:09:59.822685    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:10:00.729842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:00.729811    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:10:00.730001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:00.729863    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:10:00.831793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:00.825882    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:10:00.831793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:00.825947    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:10:01.743841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:01.743802    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:10:01.747898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:01.747858    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:10:01.826006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:01.825969    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:10:01.854944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:01.854900    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:10:02.051996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.051871    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:10:02.165090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.165047    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 23:10:02.181791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.171048    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:10:02.294409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.294320    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:02.346453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.346338    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:02.580994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.580928    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:02.618983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.618879    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:10:02.633965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.633921    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:02.749298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.742151    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:10:02.749298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.742220    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:10:02.816473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.816052    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:10:02.816473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:02.816109    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:10:03.087853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.087811    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.163365 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:10:03.163328    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:10:03.173334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.173286    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.174012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.173991    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:10:03.194987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.193864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.194987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.194279    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:10:03.214751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.214707    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.215418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.215388    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.259895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.259854    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.354855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.354163    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:10:03.354855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.354654    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:10:03.379790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.355333    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:10:03.379790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.358875    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.379790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.359355    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:10:03.639788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.634678    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.648625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.639975    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.648853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.648728    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:10:03.648853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.648834    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:10:03.845313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.830869    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:03.845313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.831882    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:10:03.845313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.831942    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:10:03.845313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.842011    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:10:03.845313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:03.842085    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:10:04.066845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.058720    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.107812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.093049    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.107812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.093486    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:10:04.193671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.188270    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.198920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.198872    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.222998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.222954    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.223745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.223723    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:10:04.260270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.260228    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.261056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.261036    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:10:04.354581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.354484    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.374760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.374731    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:10:04.375686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.354825    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:10:04.378022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.377995    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:04.648387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.648349    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:10:04.832430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.732646    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:10:04.832430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.733659    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 29 23:10:04.832430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.824354    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 29 23:10:04.833924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.833896    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:10:04.834130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:04.834108    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:10:05.100210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.100098    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:05.200789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.197960    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:10:05.239257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.225141    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:10:05.308537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.242618    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:05.637801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.262992    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 23:10:05.793979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.321058    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:05.808960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.321397    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:10:05.836029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.835989    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:10:05.837556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.439312    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:10:05.838891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.454384    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:10:05.853706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.853652    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:05.877008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.439176    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:06.889949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:05.921002    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 29 23:10:06.890640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.890605    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 29 23:10:06.985171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.244569    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.012153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.640176    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.021964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:07.014935    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:10:07.027040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.826254    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.099272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:07.099219    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:10:07.125347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.748684    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:07.127688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:07.127646    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.149470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.216473    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.154256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:07.099635    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:10:07.170861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.826301    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.243829 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:06.826369    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.244564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:07.099687    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:10:07.707790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:07.703472    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:07.915796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:07.887480    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.013849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.006242    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.022423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.022381    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.044507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.022801    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.100608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.100568    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:10:08.132822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.109951    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Oct 29 23:10:08.133111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.128951    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.133818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.130803    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:08.137788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.133987    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.204488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.155206    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.280842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.198119    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.280842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.210910    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:08.280842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.271350    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.280842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.210980    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.280842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.228880    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 23:10:08.280842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.245543    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.423792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.420540    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 29 23:10:08.606130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.606012    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:08.706743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.706691    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:10:08.715926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.715901    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:10:08.731003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.730976    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:10:08.849795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.824785    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:10:08.884439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.884179    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 29 23:10:08.950068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:08.921325    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.006804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.000040    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.046044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.045992    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.134749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.134038    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.135126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.135101    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.163666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.163628    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:09.177789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.167831    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.185821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.166234    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.186069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.166351    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.309859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.271426    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.309859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.285995    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.309859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.291290    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:10:09.309859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.295852    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:09.309859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.295917    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.504867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.502207    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:10:09.741167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.741133    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"]
Oct 29 23:10:09.845654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.844478    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.863575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.858070    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.875631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.875586    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-socket\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.878703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.878673    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d52c78-f846-4674-9556-28940d3e6c13-istiod-ca-cert\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.880134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.880110    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-istio-token\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.880409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.880391    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-data\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.880614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.880597    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-certs\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.880837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.880818    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d52c78-f846-4674-9556-28940d3e6c13-istio-podinfo\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.881141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.881124    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-credential-socket\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.881305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.881287    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkd2j\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-kube-api-access-kkd2j\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.881548 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.881530    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-envoy\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.881787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.879985    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-755453-openshift-default-dockercfg-n8z59\""
Oct 29 23:10:09.887151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.887118    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"]
Oct 29 23:10:09.976387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.959261    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:09.976387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.966260    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded"
Oct 29 23:10:09.985974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.985934    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-credential-socket\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.986866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.986840    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkd2j\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-kube-api-access-kkd2j\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.987144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.987125    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-envoy\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:09.987316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:09.987297    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-socket\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.071935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.071893    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d52c78-f846-4674-9556-28940d3e6c13-istiod-ca-cert\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.101908    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-istio-token\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.102049    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-data\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.102102    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-certs\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.102190    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d52c78-f846-4674-9556-28940d3e6c13-istio-podinfo\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.080385    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkd2j\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-kube-api-access-kkd2j\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.080754    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-envoy\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.072176    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-credential-socket\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.102834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.011502    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-socket\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.103438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.103019    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-data\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.103438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.073012    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d52c78-f846-4674-9556-28940d3e6c13-istiod-ca-cert\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.103438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.103292    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-certs\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.103438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.078953    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 29 23:10:10.107787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.104174    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:10.130321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.111018    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d52c78-f846-4674-9556-28940d3e6c13-istio-podinfo\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.171900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.171355    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:10.180800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.177739    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:10.205881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.205839    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 23:10:10.213737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.192305    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:10.213737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.192369    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:10.213737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.212324    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:10.274793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.274055    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:10:10.299789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.298856    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:10:10.305210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.301835    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:10:10.315069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.315043    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:10.315265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.315241    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:10.327484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.316321    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-istio-token\") pod \"e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") " pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.658406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.618828    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:10.853071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.853027    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:10.959677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:10.959545    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:10:11.010793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.010543    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:10:11.010793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.010628    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:10:11.304848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.239955    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:10:11.304848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.287283    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:11.305793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.305197    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:11.325795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.315831    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:10:11.338834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.329957    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:11.338834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.330351    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 23:10:11.376807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.366853    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:10:11.404716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.404663    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:11.405047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.404997    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:11.510116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.485818    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:11.510116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.485878    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:11.553672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.553641    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:10:11.572260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.572177    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:10:11.920795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.919647    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:11.941799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.923353    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:11.964029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:11.963984    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:12.259971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.255087    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:12.305834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.292092    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:10:12.305834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.292510    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:10:12.305834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.294001    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:10:12.319788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.311356    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:12.380530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.326071    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:10:12.380530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.380087    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:10:12.383137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.381878    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:10:12.383137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.382718    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:12.383137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.382932    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:12.608936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.479311    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:10:12.637600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.637561    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": context deadline exceeded"
Oct 29 23:10:12.649059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.649026    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:10:12.723788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.719292    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:10:12.931851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.907522    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:10:12.987792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.943975    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:13.007797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.971662    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:13.033196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:12.977224    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:10:13.037789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.033994    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:10:13.185089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.184988    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:13.380803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.360662    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:10:13.475838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.422370    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:13.475838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.422836    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:13.591349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.591298    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:13.591978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.591942    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:13.592184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.592165    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:13.605128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.605108    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:10:13.714141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.712939    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:13.857010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.856904    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:13.909070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.903937    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:10:13.909070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.906702    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"]
Oct 29 23:10:13.930036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.917233    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"]
Oct 29 23:10:13.930036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.923006    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:10:13.930036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.923998    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.001076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:13.992099    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:14.017257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.003220    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"]
Oct 29 23:10:14.025508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.025466    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:10:14.027352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.027309    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:10:14.048586 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.048554    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:10:14.052056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052013    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052145    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052206    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052232    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052274    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvlc\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-kube-api-access-ssvlc\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052325    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052519    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052752    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.052971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.052953    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.157383    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.157438    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.157523    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvlc\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-kube-api-access-ssvlc\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.157595    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.157836    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.157880    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.157937    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.158023    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.158239    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.159717    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.162135    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.162661    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.162966    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.163268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.163130    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.169884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.168725    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.178702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.178640    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.191694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.191662    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.194524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.194485    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvlc\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-kube-api-access-ssvlc\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.352949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.352671    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:14.408080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.406788    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" event={"ID":"c2d52c78-f846-4674-9556-28940d3e6c13","Type":"ContainerStarted","Data":"cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069"}
Oct 29 23:10:14.408080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.406852    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" event={"ID":"c2d52c78-f846-4674-9556-28940d3e6c13","Type":"ContainerStarted","Data":"3f5ca50683e3c417c8ee6789b883dddfc9a8b50870873f994769c1a570d4930d"}
Oct 29 23:10:14.458211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.458157    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podStartSLOduration=5.458135269 podStartE2EDuration="5.458135269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:14.45213891 +0000 UTC m=+1650.075049935" watchObservedRunningTime="2026-10-29 23:10:14.458135269 +0000 UTC m=+1650.081046290"
Oct 29 23:10:14.627811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.627724    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:14.744760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.744719    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"]
Oct 29 23:10:14.749035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.749002    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:14.749124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.749069    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:14.749124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:14.749097    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:15.429251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:15.429210    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" event={"ID":"d8ef02ac-a9a8-450b-8df2-739a526ca9e8","Type":"ContainerStarted","Data":"a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2"}
Oct 29 23:10:15.429755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:15.429287    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" event={"ID":"d8ef02ac-a9a8-450b-8df2-739a526ca9e8","Type":"ContainerStarted","Data":"950fb5ddc2a194629dd1230087ce1096b97cd2c45a71d448f5d1eb564182f4d4"}
Oct 29 23:10:15.506466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:15.506403    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podStartSLOduration=2.506383352 podStartE2EDuration="2.506383352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:15.485939856 +0000 UTC m=+1651.108850899" watchObservedRunningTime="2026-10-29 23:10:15.506383352 +0000 UTC m=+1651.129294373"
Oct 29 23:10:15.635885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:15.635846    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:16.354448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.354186    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:16.354448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.354433    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:10:16.354716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.354486    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:10:16.439230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.438827    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:16.441046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.441020    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:10:16.718201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.718157    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"]
Oct 29 23:10:16.727339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.727306    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.739932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.739891    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:16.740116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.739953    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:16.749855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.749822    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"]
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.759879    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760032    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760065    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760105    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760228    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760287    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh662\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-kube-api-access-fh662\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760387    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760515    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.760809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.760570    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.873569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.862290    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.873569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.862434    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.873569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.862501    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh662\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-kube-api-access-fh662\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.873569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.862596    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.873569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.862714    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.873569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.863221    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.889923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.875451    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.892339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.890552    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.892339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.890840    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.892339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.890927    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.899796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.892673    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.899796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.892883    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.899796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.892917    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:16.901193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.901135    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:17.006843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.984523    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh662\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-kube-api-access-fh662\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:17.006843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:16.984976    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:17.023793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.020131    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:17.054819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.048913    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:17.194790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.129085    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:17.356813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.356754    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:10:17.356963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.356855    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:10:17.829872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.829844    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"]
Oct 29 23:10:17.832999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.832410    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:17.832999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.832510    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:17.832999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:17.832556    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:18.353505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:18.353300    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:10:18.353505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:18.353360    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:10:18.468942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:18.468837    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" event={"ID":"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8","Type":"ContainerStarted","Data":"c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff"}
Oct 29 23:10:18.468942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:18.468898    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" event={"ID":"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8","Type":"ContainerStarted","Data":"99005227fb870e94b19d4270567b9ed73056b74d5848294064c1e4bbf0d16d5e"}
Oct 29 23:10:19.130329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:19.130294    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:19.132606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:19.131889    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Oct 29 23:10:19.132606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:19.132586    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Oct 29 23:10:19.353321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:19.353272    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:10:19.353529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:19.353370    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:10:20.131574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:20.131383    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Oct 29 23:10:20.132237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:20.132205    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Oct 29 23:10:20.354061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:20.354014    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:10:20.354714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:20.354325    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:10:20.386849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:20.380200    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:10:20.386849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:20.380262    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:21.134309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:21.134267    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Oct 29 23:10:21.134823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:21.134367    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Oct 29 23:10:21.354147 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:21.354110    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:10:21.354470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:21.354422    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:10:22.131534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:22.131195    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Oct 29 23:10:22.131534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:22.131243    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Oct 29 23:10:22.397279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:22.387903    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:22.404485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:22.403426    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:22.411308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:22.411274    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:10:22.468797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:22.468232    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podStartSLOduration=6.468210546 podStartE2EDuration="6.468210546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:18.520951528 +0000 UTC m=+1654.143862551" watchObservedRunningTime="2026-10-29 23:10:22.468210546 +0000 UTC m=+1658.091121567"
Oct 29 23:10:23.285240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.130323    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Oct 29 23:10:23.285240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.130376    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Oct 29 23:10:23.905293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.905265    4827 generic.go:356] "Generic (PLEG): container finished" podID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerID="2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c" exitCode=0
Oct 29 23:10:23.905721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.905337    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" event={"ID":"8b66c029-ab0c-4a37-82fc-889fb3667b97","Type":"ContainerDied","Data":"2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c"}
Oct 29 23:10:23.905859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.905835    4827 scope.go:122] "RemoveContainer" containerID="2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c"
Oct 29 23:10:23.907351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.907332    4827 generic.go:356] "Generic (PLEG): container finished" podID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerID="7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90" exitCode=0
Oct 29 23:10:23.907457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.907412    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" event={"ID":"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c","Type":"ContainerDied","Data":"7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90"}
Oct 29 23:10:23.908073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.907912    4827 scope.go:122] "RemoveContainer" containerID="7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90"
Oct 29 23:10:23.908184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.908074    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.908184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.908161    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.908331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.908205    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.909409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.909383    4827 generic.go:356] "Generic (PLEG): container finished" podID="b73447f3-09ac-4b57-8704-e46f8713987d" containerID="47f0cbc9ded3d9cba64bf50892d51ded48ac300df358da7ac4bbe35081439408" exitCode=0
Oct 29 23:10:23.909640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.909615    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.909694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.909680    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.909735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.909721    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.909870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.909852    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" event={"ID":"b73447f3-09ac-4b57-8704-e46f8713987d","Type":"ContainerDied","Data":"47f0cbc9ded3d9cba64bf50892d51ded48ac300df358da7ac4bbe35081439408"}
Oct 29 23:10:23.910225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.910210    4827 scope.go:122] "RemoveContainer" containerID="47f0cbc9ded3d9cba64bf50892d51ded48ac300df358da7ac4bbe35081439408"
Oct 29 23:10:23.912383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.912350    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.912474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.912428    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:23.912547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:23.912477    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:24.130270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:24.130173    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Oct 29 23:10:24.130270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:24.130221    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Oct 29 23:10:25.024790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.024715    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" event={"ID":"8b66c029-ab0c-4a37-82fc-889fb3667b97","Type":"ContainerStarted","Data":"27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd"}
Oct 29 23:10:25.046789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.034185    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" event={"ID":"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c","Type":"ContainerStarted","Data":"54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c"}
Oct 29 23:10:25.046789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.034243    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" event={"ID":"b73447f3-09ac-4b57-8704-e46f8713987d","Type":"ContainerStarted","Data":"ab5627e2ad70b44e676bc83476fef547ead438b83d70183cfccb739517d49fd3"}
Oct 29 23:10:25.126789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.126118    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:10:25.197790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.180194    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 29 23:10:25.197790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.180257    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 29 23:10:25.274849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.274621    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:10:25.274849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.274750    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 23:10:25.288581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.282853    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 23:10:25.471789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.468133    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:25.537883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.498802    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:25.562159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.547121    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:10:25.970453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.938261    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:25.970453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.950483    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:10:25.989927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.989427    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 23:10:25.989927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:25.989490    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 23:10:26.118793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.118539    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:26.151845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.147526    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 29 23:10:26.151845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.147583    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 29 23:10:26.198413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.198376    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:26.237837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.233122    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:26.287952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.287917    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 23:10:26.304885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.288190    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 23:10:26.468593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.467905    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:26.643975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.643942    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:10:26.952053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.951898    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 23:10:26.952305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:26.952276    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 23:10:27.146838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:27.144958    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 29 23:10:27.146838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:27.145029    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 29 23:10:27.289846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:27.289135    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 23:10:27.289846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:27.289194    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 23:10:27.947155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:27.947118    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 23:10:27.947462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:27.947434    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 23:10:28.126789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:28.119132    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 29 23:10:28.126789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:28.119192    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 29 23:10:28.289064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:28.288979    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 23:10:28.289715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:28.289683    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 23:10:28.953565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:28.951305    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 23:10:28.953565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:28.951382    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 23:10:29.129230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:29.129197    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 29 23:10:29.129523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:29.129493    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 29 23:10:29.284848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:29.279080    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 23:10:29.284848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:29.279632    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 23:10:29.942738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:29.942702    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 23:10:29.943371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:29.943341    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 23:10:30.119791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:30.119082    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 29 23:10:30.119791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:30.119139    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 29 23:10:30.310843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:30.291116    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 23:10:30.310843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:30.291200    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 23:10:30.698788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:30.697875    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:10:30.947789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:30.946866    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 23:10:30.947789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:30.946925    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 23:10:31.122849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.122124    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 29 23:10:31.122849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.122186    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 29 23:10:31.275460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.275423    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 23:10:31.281906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.281867    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 23:10:31.344794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.344653    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"]
Oct 29 23:10:31.354818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.353749    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"]
Oct 29 23:10:31.354818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.353915    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.407851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.402485    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-65e2d3-serving-cert\""
Oct 29 23:10:31.503646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.503611    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.503871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.503841    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.504020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.503999    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvnw\" (UniqueName: \"kubernetes.io/projected/88837411-4699-4681-8ba8-7a8eada3293f-kube-api-access-crvnw\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.504177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.504161    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88837411-4699-4681-8ba8-7a8eada3293f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.610591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.610550    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.613048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.612997    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.613927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.613904    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvnw\" (UniqueName: \"kubernetes.io/projected/88837411-4699-4681-8ba8-7a8eada3293f-kube-api-access-crvnw\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.614714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.614679    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88837411-4699-4681-8ba8-7a8eada3293f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.615716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.615502    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88837411-4699-4681-8ba8-7a8eada3293f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.641088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.641056    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.652852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.643475    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.653103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.646620    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvnw\" (UniqueName: \"kubernetes.io/projected/88837411-4699-4681-8ba8-7a8eada3293f-kube-api-access-crvnw\") pod \"maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.734989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.733818    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:31.954792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.954741    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:10:31.957403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:31.957379    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:10:32.119288 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:10:32.119258    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88837411_4699_4681_8ba8_7a8eada3293f.slice/crio-41c260c7575d7e9cceb0e7f3a2c2bc5ff6e545e56ffef2d16e9529c74a1211f2 WatchSource:0}: Error finding container 41c260c7575d7e9cceb0e7f3a2c2bc5ff6e545e56ffef2d16e9529c74a1211f2: Status 404 returned error can't find the container with id 41c260c7575d7e9cceb0e7f3a2c2bc5ff6e545e56ffef2d16e9529c74a1211f2
Oct 29 23:10:32.120076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:32.119951    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"]
Oct 29 23:10:32.124415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:32.124389    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:10:32.125044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:32.125023    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:10:32.125863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:32.125843    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:10:32.278740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:32.278711    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:10:32.280031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:32.280006    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:10:32.439486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:32.439446    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj" event={"ID":"88837411-4699-4681-8ba8-7a8eada3293f","Type":"ContainerStarted","Data":"41c260c7575d7e9cceb0e7f3a2c2bc5ff6e545e56ffef2d16e9529c74a1211f2"}
Oct 29 23:10:32.836647 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:10:32.836559    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:10:33.461539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.461498    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj" event={"ID":"88837411-4699-4681-8ba8-7a8eada3293f","Type":"ContainerStarted","Data":"a69107d273f0731699cce9cf008e17f8ad2e0571eeede0f68350f0b131d51b56"}
Oct 29 23:10:33.462050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.461584    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:33.515625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.511687    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj" podStartSLOduration=2.20775012 podStartE2EDuration="2.511663663s" totalImagesPullingTime="303.913543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:33.507946063 +0000 UTC m=+1669.130857083" watchObservedRunningTime="2026-10-29 23:10:33.511663663 +0000 UTC m=+1669.134574684"
Oct 29 23:10:33.850648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.850572    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"]
Oct 29 23:10:33.854424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.854397    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.881694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.881667    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"]
Oct 29 23:10:33.983842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.969188    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.983842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.969273    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.983842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.969315    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.983842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.969350    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.983842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.969436    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.983842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.969568    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea8da08-ef56-4327-becb-86c5dff41b70-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.983842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.969616    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.994786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.991045    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:33.994786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:33.991184    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62qwl\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-kube-api-access-62qwl\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.017797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.015037    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ed3bf2-openshift-default-dockercfg-tnc4j\""
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094082    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094204    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62qwl\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-kube-api-access-62qwl\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094254    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094331    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094518    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094556    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094641    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094802    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea8da08-ef56-4327-becb-86c5dff41b70-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.094848    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.101869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.095820    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.116788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.114208    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.116788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.114459    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.116788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.115170    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.116788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.116122    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea8da08-ef56-4327-becb-86c5dff41b70-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.150315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.150286    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.162803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.160166    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.162803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.160745    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.162803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.161899    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62qwl\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-kube-api-access-62qwl\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.178809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.178070    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:34.782439 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:10:34.782408    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea8da08_ef56_4327_becb_86c5dff41b70.slice/crio-0257317de01450eace9696e1c06676b21a8cd056b786cc0d2988f874c9aed309 WatchSource:0}: Error finding container 0257317de01450eace9696e1c06676b21a8cd056b786cc0d2988f874c9aed309: Status 404 returned error can't find the container with id 0257317de01450eace9696e1c06676b21a8cd056b786cc0d2988f874c9aed309
Oct 29 23:10:34.782991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.782970    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"]
Oct 29 23:10:34.785475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.785438    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:34.785571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.785508    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:34.785571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:34.785538    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:35.523981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:35.523941    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" event={"ID":"8ea8da08-ef56-4327-becb-86c5dff41b70","Type":"ContainerStarted","Data":"722f3586bcc8062d906c98c8c71c5f4ec14d332cbdc8669e7781ff552f5021ca"}
Oct 29 23:10:35.524199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:35.524010    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" event={"ID":"8ea8da08-ef56-4327-becb-86c5dff41b70","Type":"ContainerStarted","Data":"0257317de01450eace9696e1c06676b21a8cd056b786cc0d2988f874c9aed309"}
Oct 29 23:10:35.598557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:35.598506    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podStartSLOduration=2.598488034 podStartE2EDuration="2.598488034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:35.592167338 +0000 UTC m=+1671.215078357" watchObservedRunningTime="2026-10-29 23:10:35.598488034 +0000 UTC m=+1671.221399056"
Oct 29 23:10:36.178747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:36.178711    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:36.185191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:36.185167    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:36.531354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:36.531274    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:36.532276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:36.532257    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:10:40.852422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:40.852390    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:10:50.046478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:50.046444    4827 generic.go:356] "Generic (PLEG): container finished" podID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerID="e3a2070c02710eca274382963ceff2c14f999130a263c9d8b9f21ec14e8a7a81" exitCode=0
Oct 29 23:10:50.046924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:50.046516    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" event={"ID":"d06c4719-725f-4a22-8359-8d5d2de03a60","Type":"ContainerDied","Data":"e3a2070c02710eca274382963ceff2c14f999130a263c9d8b9f21ec14e8a7a81"}
Oct 29 23:10:50.047029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:50.047014    4827 scope.go:122] "RemoveContainer" containerID="e3a2070c02710eca274382963ceff2c14f999130a263c9d8b9f21ec14e8a7a81"
Oct 29 23:10:50.048847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:50.048818    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:50.048940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:50.048893    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:50.048940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:50.048927    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:10:51.104334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:51.104298    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" event={"ID":"d06c4719-725f-4a22-8359-8d5d2de03a60","Type":"ContainerStarted","Data":"7b1af9d45ddba4568f35fb8bbe7f5d458639ca64fedd0b2ad8768a7137b23636"}
Oct 29 23:10:51.321587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:51.321553    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:10:51.323170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:51.323144    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 23:10:51.323313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:51.323192    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 23:10:52.322186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:52.322150    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 23:10:52.322689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:52.322204    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 23:10:53.321476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:53.321444    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 23:10:53.321656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:53.321502    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 23:10:54.321693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:54.321661    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 23:10:54.322281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:54.321734    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 23:10:55.209650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:55.209621    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:10:55.322222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:55.322188    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 23:10:55.322703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:55.322246    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 23:10:56.322353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:56.322321    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 23:10:56.322752 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:56.322378    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 23:10:57.326703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:57.326667    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:10:57.328066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:10:57.328040    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:11:02.710759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.710725    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"]
Oct 29 23:11:02.745801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.717339    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"]
Oct 29 23:11:02.767455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.767410    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:02.797006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.795189    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:02.797006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.796216    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:11:02.900788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.853973    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:02.900788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.854295    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:02.900788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.854408    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89jz\" (UniqueName: \"kubernetes.io/projected/23dab02e-e105-46a1-b2cb-560a173c3159-kube-api-access-v89jz\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:02.990252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.990156    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89jz\" (UniqueName: \"kubernetes.io/projected/23dab02e-e105-46a1-b2cb-560a173c3159-kube-api-access-v89jz\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:02.995295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.995265    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:02.997055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.997030    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:02.997870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:02.996220    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:03.014269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:03.014231    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:03.149809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:03.146558    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89jz\" (UniqueName: \"kubernetes.io/projected/23dab02e-e105-46a1-b2cb-560a173c3159-kube-api-access-v89jz\") pod \"maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:03.165855 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:03.165636    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:11:03.190793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:03.189937    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:03.556689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:03.556660    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"]
Oct 29 23:11:03.557629 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:11:03.557477    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dab02e_e105_46a1_b2cb_560a173c3159.slice/crio-14877ab306d7444dcef9e969a6f202a3b56f5b13fa75b9f959b10b9470990783 WatchSource:0}: Error finding container 14877ab306d7444dcef9e969a6f202a3b56f5b13fa75b9f959b10b9470990783: Status 404 returned error can't find the container with id 14877ab306d7444dcef9e969a6f202a3b56f5b13fa75b9f959b10b9470990783
Oct 29 23:11:04.539909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:04.539881    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9" event={"ID":"23dab02e-e105-46a1-b2cb-560a173c3159","Type":"ContainerStarted","Data":"5fe0a02edc0589df8ccebf426229451932bc43fcf9609d6cde236b79a8e5f595"}
Oct 29 23:11:04.540316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:04.539929    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9" event={"ID":"23dab02e-e105-46a1-b2cb-560a173c3159","Type":"ContainerStarted","Data":"14877ab306d7444dcef9e969a6f202a3b56f5b13fa75b9f959b10b9470990783"}
Oct 29 23:11:04.581276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:04.581213    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9" podStartSLOduration=2.581194321 podStartE2EDuration="2.581194321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:04.578850704 +0000 UTC m=+1700.201761725" watchObservedRunningTime="2026-10-29 23:11:04.581194321 +0000 UTC m=+1700.204105342"
Oct 29 23:11:05.667195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:05.666589    4827 generic.go:356] "Generic (PLEG): container finished" podID="23dab02e-e105-46a1-b2cb-560a173c3159" containerID="5fe0a02edc0589df8ccebf426229451932bc43fcf9609d6cde236b79a8e5f595" exitCode=0
Oct 29 23:11:05.667195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:05.666717    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9" event={"ID":"23dab02e-e105-46a1-b2cb-560a173c3159","Type":"ContainerDied","Data":"5fe0a02edc0589df8ccebf426229451932bc43fcf9609d6cde236b79a8e5f595"}
Oct 29 23:11:06.871156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.871129    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:06.951625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.951535    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23dab02e-e105-46a1-b2cb-560a173c3159-kube-api-access-v89jz\" (UniqueName: \"kubernetes.io/projected/23dab02e-e105-46a1-b2cb-560a173c3159-kube-api-access-v89jz\") pod \"23dab02e-e105-46a1-b2cb-560a173c3159\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") "
Oct 29 23:11:06.951625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.951586    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-xks-ca\") pod \"23dab02e-e105-46a1-b2cb-560a173c3159\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") "
Oct 29 23:11:06.951881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.951629    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-openshift-ca\") pod \"23dab02e-e105-46a1-b2cb-560a173c3159\" (UID: \"23dab02e-e105-46a1-b2cb-560a173c3159\") "
Oct 29 23:11:06.952254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.952201    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-openshift-ca" pod "23dab02e-e105-46a1-b2cb-560a173c3159" (UID: "23dab02e-e105-46a1-b2cb-560a173c3159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:06.952681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.952655    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:06.954950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.954921    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dab02e-e105-46a1-b2cb-560a173c3159-kube-api-access-v89jz" pod "23dab02e-e105-46a1-b2cb-560a173c3159" (UID: "23dab02e-e105-46a1-b2cb-560a173c3159"). InnerVolumeSpecName "kube-api-access-v89jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:06.955221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:06.955177    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-xks-ca" pod "23dab02e-e105-46a1-b2cb-560a173c3159" (UID: "23dab02e-e105-46a1-b2cb-560a173c3159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:07.054449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:07.054414    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89jz\" (UniqueName: \"kubernetes.io/projected/23dab02e-e105-46a1-b2cb-560a173c3159-kube-api-access-v89jz\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:07.054449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:07.054449    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23dab02e-e105-46a1-b2cb-560a173c3159-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:07.684377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:07.684353    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"
Oct 29 23:11:07.684622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:07.684352    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9" event={"ID":"23dab02e-e105-46a1-b2cb-560a173c3159","Type":"ContainerDied","Data":"14877ab306d7444dcef9e969a6f202a3b56f5b13fa75b9f959b10b9470990783"}
Oct 29 23:11:07.684622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:07.684486    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14877ab306d7444dcef9e969a6f202a3b56f5b13fa75b9f959b10b9470990783"
Oct 29 23:11:09.766228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.766197    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"]
Oct 29 23:11:09.767001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.766984    4827 cpu_manager.go:414] "containerMap: removing container" podUID="23dab02e-e105-46a1-b2cb-560a173c3159" containerName="revoke-keys"
Oct 29 23:11:09.767079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.767001    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dab02e-e105-46a1-b2cb-560a173c3159" containerName="revoke-keys"
Oct 29 23:11:09.767132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.767123    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dab02e-e105-46a1-b2cb-560a173c3159" containerName="revoke-keys"
Oct 29 23:11:09.770906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.770888    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.785869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.785848    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ed3bf2-serving-cert\""
Oct 29 23:11:09.803199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.803172    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"]
Oct 29 23:11:09.852540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.852507    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d61656d-824f-4b95-b46e-7d7d4cec7645-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.852745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.852559    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwms6\" (UniqueName: \"kubernetes.io/projected/3d61656d-824f-4b95-b46e-7d7d4cec7645-kube-api-access-gwms6\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.852852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.852811    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.852960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.852941    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.955240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.955201    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d61656d-824f-4b95-b46e-7d7d4cec7645-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.955440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.955271    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwms6\" (UniqueName: \"kubernetes.io/projected/3d61656d-824f-4b95-b46e-7d7d4cec7645-kube-api-access-gwms6\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.955440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.955424    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.955563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.955526    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.959917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.956431    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d61656d-824f-4b95-b46e-7d7d4cec7645-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.975123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.975092    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.975274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.975247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:09.978746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:09.978718    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwms6\" (UniqueName: \"kubernetes.io/projected/3d61656d-824f-4b95-b46e-7d7d4cec7645-kube-api-access-gwms6\") pod \"maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:10.089846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:10.085111    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:10.517522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:10.517492    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"]
Oct 29 23:11:10.519194 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:11:10.519168    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d61656d_824f_4b95_b46e_7d7d4cec7645.slice/crio-c92b33bd05cb42d89a4c979abbe5fdc2f99e2a8ff82e8ed6d98d407ee2049a1c WatchSource:0}: Error finding container c92b33bd05cb42d89a4c979abbe5fdc2f99e2a8ff82e8ed6d98d407ee2049a1c: Status 404 returned error can't find the container with id c92b33bd05cb42d89a4c979abbe5fdc2f99e2a8ff82e8ed6d98d407ee2049a1c
Oct 29 23:11:10.713897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:10.713863    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t" event={"ID":"3d61656d-824f-4b95-b46e-7d7d4cec7645","Type":"ContainerStarted","Data":"c92b33bd05cb42d89a4c979abbe5fdc2f99e2a8ff82e8ed6d98d407ee2049a1c"}
Oct 29 23:11:11.785586 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:11.785549    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t" event={"ID":"3d61656d-824f-4b95-b46e-7d7d4cec7645","Type":"ContainerStarted","Data":"36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64"}
Oct 29 23:11:11.787547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:11.787521    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:11.856235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:11.856156    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t" podStartSLOduration=2.4248051 podStartE2EDuration="2.856138822s" totalImagesPullingTime="431.333722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:11.849539307 +0000 UTC m=+1707.472450327" watchObservedRunningTime="2026-10-29 23:11:11.856138822 +0000 UTC m=+1707.479049840"
Oct 29 23:11:12.882557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:12.882491    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"]
Oct 29 23:11:12.917084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:12.917056    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"]
Oct 29 23:11:17.883200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:17.883128    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn" podUID="f63e8edd-f701-4d64-aa15-ce42ad66743d" containerName="payload-processing" containerID="cri-o://abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b" gracePeriod=25
Oct 29 23:11:17.917625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:17.917592    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc" podUID="f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" containerName="payload-pre-processing" containerID="cri-o://2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2" gracePeriod=25
Oct 29 23:11:18.344675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.344647    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:11:18.388462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.388430    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:11:18.504379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.504307    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63e8edd-f701-4d64-aa15-ce42ad66743d-kube-api-access-9vlrg\" (UniqueName: \"kubernetes.io/projected/f63e8edd-f701-4d64-aa15-ce42ad66743d-kube-api-access-9vlrg\") pod \"f63e8edd-f701-4d64-aa15-ce42ad66743d\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") "
Oct 29 23:11:18.504532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.504498    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-plugins-config-volume\") pod \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") "
Oct 29 23:11:18.504641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.504625    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63e8edd-f701-4d64-aa15-ce42ad66743d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63e8edd-f701-4d64-aa15-ce42ad66743d-plugins-config-volume\") pod \"f63e8edd-f701-4d64-aa15-ce42ad66743d\" (UID: \"f63e8edd-f701-4d64-aa15-ce42ad66743d\") "
Oct 29 23:11:18.504724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.504658    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-kube-api-access-dp4sx\" (UniqueName: \"kubernetes.io/projected/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-kube-api-access-dp4sx\") pod \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\" (UID: \"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4\") "
Oct 29 23:11:18.504993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.504967    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-plugins-config-volume" pod "f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" (UID: "f9fa0658-f0c4-47c1-ac79-4c131e96b8a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:18.505093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.505026    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63e8edd-f701-4d64-aa15-ce42ad66743d-plugins-config-volume" pod "f63e8edd-f701-4d64-aa15-ce42ad66743d" (UID: "f63e8edd-f701-4d64-aa15-ce42ad66743d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:18.505399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.505376    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:18.505399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.505398    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f63e8edd-f701-4d64-aa15-ce42ad66743d-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:18.506476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.506456    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63e8edd-f701-4d64-aa15-ce42ad66743d-kube-api-access-9vlrg" pod "f63e8edd-f701-4d64-aa15-ce42ad66743d" (UID: "f63e8edd-f701-4d64-aa15-ce42ad66743d"). InnerVolumeSpecName "kube-api-access-9vlrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:18.506843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.506817    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-kube-api-access-dp4sx" pod "f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" (UID: "f9fa0658-f0c4-47c1-ac79-4c131e96b8a4"). InnerVolumeSpecName "kube-api-access-dp4sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:18.607987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.607946    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4sx\" (UniqueName: \"kubernetes.io/projected/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4-kube-api-access-dp4sx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:18.607987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.607987    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vlrg\" (UniqueName: \"kubernetes.io/projected/f63e8edd-f701-4d64-aa15-ce42ad66743d-kube-api-access-9vlrg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:18.848994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.848926    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:11:18.892032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.892000    4827 generic.go:356] "Generic (PLEG): container finished" podID="f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" containerID="2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2" exitCode=0
Oct 29 23:11:18.892488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.892067    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"
Oct 29 23:11:18.892488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.892102    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc" event={"ID":"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4","Type":"ContainerDied","Data":"2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2"}
Oct 29 23:11:18.892488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.892155    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc" event={"ID":"f9fa0658-f0c4-47c1-ac79-4c131e96b8a4","Type":"ContainerDied","Data":"a793ee7b0a8b61829cd7842fce8c5a3bbd835248c68c6bfc454d9437033adbaf"}
Oct 29 23:11:18.892488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.892184    4827 scope.go:122] "RemoveContainer" containerID="2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2"
Oct 29 23:11:18.894137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.894110    4827 generic.go:356] "Generic (PLEG): container finished" podID="f63e8edd-f701-4d64-aa15-ce42ad66743d" containerID="abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b" exitCode=0
Oct 29 23:11:18.894304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.894164    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn" event={"ID":"f63e8edd-f701-4d64-aa15-ce42ad66743d","Type":"ContainerDied","Data":"abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b"}
Oct 29 23:11:18.894304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.894184    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"
Oct 29 23:11:18.894304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.894209    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn" event={"ID":"f63e8edd-f701-4d64-aa15-ce42ad66743d","Type":"ContainerDied","Data":"7d4434525524edb8431f1095ae834854833cce1e56fce05e69354fc607542aef"}
Oct 29 23:11:18.913063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.913042    4827 scope.go:122] "RemoveContainer" containerID="2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2"
Oct 29 23:11:18.913368 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:18.913346    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2\": container with ID starting with 2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2 not found: ID does not exist" containerID="2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2"
Oct 29 23:11:18.913422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.913380    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2"} err="failed to get container status \"2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2\": rpc error: code = NotFound desc = could not find container \"2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2\": container with ID starting with 2faae5cfc32bf5ab25a0a473c79a702c01020da1ec791271ea2791df40f45ef2 not found: ID does not exist"
Oct 29 23:11:18.913422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.913404    4827 scope.go:122] "RemoveContainer" containerID="abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b"
Oct 29 23:11:18.925978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.925959    4827 scope.go:122] "RemoveContainer" containerID="abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b"
Oct 29 23:11:18.926350 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:18.926321    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b\": container with ID starting with abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b not found: ID does not exist" containerID="abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b"
Oct 29 23:11:18.926424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.926350    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b"} err="failed to get container status \"abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b\": rpc error: code = NotFound desc = could not find container \"abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b\": container with ID starting with abf7bb818cba27ab6510beb277a19f53a491c50486a06e61aebb632fd2003b7b not found: ID does not exist"
Oct 29 23:11:18.978363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.978329    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"]
Oct 29 23:11:18.985814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:18.985780    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a9ced3-796dbc46c5-kq8tc"]
Oct 29 23:11:19.017799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:19.015429    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"]
Oct 29 23:11:19.020075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:19.020050    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a9ced3-5ff6d45498-8mlcn"]
Oct 29 23:11:20.490870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:20.490840    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f63e8edd-f701-4d64-aa15-ce42ad66743d" path="/var/lib/kubelet/pods/f63e8edd-f701-4d64-aa15-ce42ad66743d/volumes"
Oct 29 23:11:20.491310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:20.491296    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" path="/var/lib/kubelet/pods/f9fa0658-f0c4-47c1-ac79-4c131e96b8a4/volumes"
Oct 29 23:11:22.433966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.433928    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"]
Oct 29 23:11:22.434882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.434858    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f63e8edd-f701-4d64-aa15-ce42ad66743d" containerName="payload-processing"
Oct 29 23:11:22.434882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.434882    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63e8edd-f701-4d64-aa15-ce42ad66743d" containerName="payload-processing"
Oct 29 23:11:22.435061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.434941    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" containerName="payload-pre-processing"
Oct 29 23:11:22.435061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.434950    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" containerName="payload-pre-processing"
Oct 29 23:11:22.435170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.435111    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9fa0658-f0c4-47c1-ac79-4c131e96b8a4" containerName="payload-pre-processing"
Oct 29 23:11:22.435170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.435130    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63e8edd-f701-4d64-aa15-ce42ad66743d" containerName="payload-processing"
Oct 29 23:11:22.440839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.440817    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.457981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.457960    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-b8f5c4-openshift-default-dockercfg-q74s6\""
Oct 29 23:11:22.477596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.477568    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"]
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591017    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-socket\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591154    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-token\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591291    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-credential-socket\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591580    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjbx\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-kube-api-access-2fjbx\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591669    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79663068-a8f3-4945-bcc6-fba2936cd3c5-istiod-ca-cert\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591699    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-certs\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591726    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-podinfo\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.591848    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-data\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.613860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.605991    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-envoy\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.713149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.713050    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-credential-socket\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.713554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.713534    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjbx\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-kube-api-access-2fjbx\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.713699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.713684    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79663068-a8f3-4945-bcc6-fba2936cd3c5-istiod-ca-cert\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.713836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.713820    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-certs\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.713958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.713943    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-podinfo\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.714153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.714135    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-data\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.714381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.714364    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-envoy\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.714524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.714508    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-socket\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.714669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.714654    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-token\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.715605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.715572    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-credential-socket\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.716644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.716621    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79663068-a8f3-4945-bcc6-fba2936cd3c5-istiod-ca-cert\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.717057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.717036    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-certs\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.728506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.728449    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-podinfo\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.728970 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.728948    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-data\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.729701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.729678    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-socket\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.739089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.739060    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-envoy\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.763731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.763699    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-token\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:22.778791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:22.776680    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjbx\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-kube-api-access-2fjbx\") pod \"e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") " pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:23.069241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:23.068146    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:23.140492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:23.131577    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"]
Oct 29 23:11:23.140492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:23.131994    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr" podUID="d59d5004-e05b-495e-ad82-fb7d0c5dfa66" containerName="maas-api" containerID="cri-o://b79d570e24082d1591a994b47a81c0a27583e27e77327871a7865e80f6b04372" gracePeriod=30
Oct 29 23:11:23.676939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:23.676895    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"]
Oct 29 23:11:23.777938 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:11:23.771050    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79663068_a8f3_4945_bcc6_fba2936cd3c5.slice/crio-c889f9e8cd28ff03cd0b055c28a110b890afe079a09f958f45b9b767327a3314 WatchSource:0}: Error finding container c889f9e8cd28ff03cd0b055c28a110b890afe079a09f958f45b9b767327a3314: Status 404 returned error can't find the container with id c889f9e8cd28ff03cd0b055c28a110b890afe079a09f958f45b9b767327a3314
Oct 29 23:11:23.800964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:23.796689    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:23.800964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:23.796819    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:23.800964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:23.796871    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:24.039248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.039217    4827 generic.go:356] "Generic (PLEG): container finished" podID="d59d5004-e05b-495e-ad82-fb7d0c5dfa66" containerID="b79d570e24082d1591a994b47a81c0a27583e27e77327871a7865e80f6b04372" exitCode=0
Oct 29 23:11:24.039424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.039405    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr" event={"ID":"d59d5004-e05b-495e-ad82-fb7d0c5dfa66","Type":"ContainerDied","Data":"b79d570e24082d1591a994b47a81c0a27583e27e77327871a7865e80f6b04372"}
Oct 29 23:11:24.041137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.041060    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" event={"ID":"79663068-a8f3-4945-bcc6-fba2936cd3c5","Type":"ContainerStarted","Data":"c889f9e8cd28ff03cd0b055c28a110b890afe079a09f958f45b9b767327a3314"}
Oct 29 23:11:24.226664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.226627    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"]
Oct 29 23:11:24.237856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.237829    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a9ced3-jk7k9"]
Oct 29 23:11:24.272494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.272471    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:11:24.392574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.392544    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-kube-api-access-ljd9n\" (UniqueName: \"kubernetes.io/projected/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-kube-api-access-ljd9n\") pod \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") "
Oct 29 23:11:24.392810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.392670    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-metrics-certs\") pod \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") "
Oct 29 23:11:24.392810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.392783    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-maas-api-tls\") pod \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") "
Oct 29 23:11:24.392961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.392874    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-openshift-service-ca\") pod \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\" (UID: \"d59d5004-e05b-495e-ad82-fb7d0c5dfa66\") "
Oct 29 23:11:24.393585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.393554    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-openshift-service-ca" pod "d59d5004-e05b-495e-ad82-fb7d0c5dfa66" (UID: "d59d5004-e05b-495e-ad82-fb7d0c5dfa66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:24.396158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.396133    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-kube-api-access-ljd9n" pod "d59d5004-e05b-495e-ad82-fb7d0c5dfa66" (UID: "d59d5004-e05b-495e-ad82-fb7d0c5dfa66"). InnerVolumeSpecName "kube-api-access-ljd9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:24.396262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.396207    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-metrics-certs" pod "d59d5004-e05b-495e-ad82-fb7d0c5dfa66" (UID: "d59d5004-e05b-495e-ad82-fb7d0c5dfa66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:24.396363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.396337    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-maas-api-tls" pod "d59d5004-e05b-495e-ad82-fb7d0c5dfa66" (UID: "d59d5004-e05b-495e-ad82-fb7d0c5dfa66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:24.540800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.523978    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:24.540800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.524015    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljd9n\" (UniqueName: \"kubernetes.io/projected/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-kube-api-access-ljd9n\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:24.540800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.524031    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:24.540800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.524044    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59d5004-e05b-495e-ad82-fb7d0c5dfa66-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:24.604741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:24.602783    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23dab02e-e105-46a1-b2cb-560a173c3159" path="/var/lib/kubelet/pods/23dab02e-e105-46a1-b2cb-560a173c3159/volumes"
Oct 29 23:11:25.051747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.051648    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" event={"ID":"79663068-a8f3-4945-bcc6-fba2936cd3c5","Type":"ContainerStarted","Data":"06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099"}
Oct 29 23:11:25.053206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.053182    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr" event={"ID":"d59d5004-e05b-495e-ad82-fb7d0c5dfa66","Type":"ContainerDied","Data":"b7ee6fcc122239cf7af9eb31b46a4116b3fd5b9957ce5e4cb057a586fb605436"}
Oct 29 23:11:25.053206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.053203    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"
Oct 29 23:11:25.053389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.053229    4827 scope.go:122] "RemoveContainer" containerID="b79d570e24082d1591a994b47a81c0a27583e27e77327871a7865e80f6b04372"
Oct 29 23:11:25.070202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.070180    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:25.076861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.076840    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:25.114062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.114014    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podStartSLOduration=3.114001573 podStartE2EDuration="3.114001573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:25.113410123 +0000 UTC m=+1720.736321144" watchObservedRunningTime="2026-10-29 23:11:25.114001573 +0000 UTC m=+1720.736912593"
Oct 29 23:11:25.196397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.196366    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"]
Oct 29 23:11:25.202341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.202315    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a9ced3-b65dd5f97-p8ptr"]
Oct 29 23:11:25.884928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.884896    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"]
Oct 29 23:11:25.885920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.885896    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d59d5004-e05b-495e-ad82-fb7d0c5dfa66" containerName="maas-api"
Oct 29 23:11:25.885920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.885919    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59d5004-e05b-495e-ad82-fb7d0c5dfa66" containerName="maas-api"
Oct 29 23:11:25.886175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.886099    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59d5004-e05b-495e-ad82-fb7d0c5dfa66" containerName="maas-api"
Oct 29 23:11:25.891345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.891323    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:25.909049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.909026    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"]
Oct 29 23:11:25.914922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.914897    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:25.915085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.915021    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:11:25.976882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.976832    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:25.977203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.977184    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:25.977250 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:25.977227    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz5xc\" (UniqueName: \"kubernetes.io/projected/8b49b552-73e1-413d-a84c-b234f96289b9-kube-api-access-xz5xc\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.063827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.063797    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:26.065642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.065615    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:11:26.079020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.078994    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.079160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.079038    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz5xc\" (UniqueName: \"kubernetes.io/projected/8b49b552-73e1-413d-a84c-b234f96289b9-kube-api-access-xz5xc\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.079160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.079092    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.079737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.079717    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.081335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.081318    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.102474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.102449    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz5xc\" (UniqueName: \"kubernetes.io/projected/8b49b552-73e1-413d-a84c-b234f96289b9-kube-api-access-xz5xc\") pod \"maas-api-revoke-sub-c8be0dd7-8v4c9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.209859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.209309    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:26.564451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.564313    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59d5004-e05b-495e-ad82-fb7d0c5dfa66" path="/var/lib/kubelet/pods/d59d5004-e05b-495e-ad82-fb7d0c5dfa66/volumes"
Oct 29 23:11:26.885735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:26.885702    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"]
Oct 29 23:11:26.886606 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:11:26.886577    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b49b552_73e1_413d_a84c_b234f96289b9.slice/crio-4f5363eaa8b3b053c680301831c4f9226d0533f83c7ff4f5b6aadee587a36539 WatchSource:0}: Error finding container 4f5363eaa8b3b053c680301831c4f9226d0533f83c7ff4f5b6aadee587a36539: Status 404 returned error can't find the container with id 4f5363eaa8b3b053c680301831c4f9226d0533f83c7ff4f5b6aadee587a36539
Oct 29 23:11:27.071907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:27.071869    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9" event={"ID":"8b49b552-73e1-413d-a84c-b234f96289b9","Type":"ContainerStarted","Data":"4f5363eaa8b3b053c680301831c4f9226d0533f83c7ff4f5b6aadee587a36539"}
Oct 29 23:11:28.081435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:28.081351    4827 generic.go:356] "Generic (PLEG): container finished" podID="8b49b552-73e1-413d-a84c-b234f96289b9" containerID="026491fcd55ee40dc78c2a8176284053d6e5320786a13193345219bed39e4c63" exitCode=0
Oct 29 23:11:28.081842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:28.081444    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9" event={"ID":"8b49b552-73e1-413d-a84c-b234f96289b9","Type":"ContainerDied","Data":"026491fcd55ee40dc78c2a8176284053d6e5320786a13193345219bed39e4c63"}
Oct 29 23:11:29.301321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.301289    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:29.353239 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.353150    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-openshift-ca\") pod \"8b49b552-73e1-413d-a84c-b234f96289b9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") "
Oct 29 23:11:29.353412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.353254    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b49b552-73e1-413d-a84c-b234f96289b9-kube-api-access-xz5xc\" (UniqueName: \"kubernetes.io/projected/8b49b552-73e1-413d-a84c-b234f96289b9-kube-api-access-xz5xc\") pod \"8b49b552-73e1-413d-a84c-b234f96289b9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") "
Oct 29 23:11:29.353412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.353397    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-xks-ca\") pod \"8b49b552-73e1-413d-a84c-b234f96289b9\" (UID: \"8b49b552-73e1-413d-a84c-b234f96289b9\") "
Oct 29 23:11:29.353529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.353433    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-openshift-ca" pod "8b49b552-73e1-413d-a84c-b234f96289b9" (UID: "8b49b552-73e1-413d-a84c-b234f96289b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:29.354182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.354156    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:29.355610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.355572    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b49b552-73e1-413d-a84c-b234f96289b9-kube-api-access-xz5xc" pod "8b49b552-73e1-413d-a84c-b234f96289b9" (UID: "8b49b552-73e1-413d-a84c-b234f96289b9"). InnerVolumeSpecName "kube-api-access-xz5xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:29.355610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.355582    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-xks-ca" pod "8b49b552-73e1-413d-a84c-b234f96289b9" (UID: "8b49b552-73e1-413d-a84c-b234f96289b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:29.456247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.456214    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz5xc\" (UniqueName: \"kubernetes.io/projected/8b49b552-73e1-413d-a84c-b234f96289b9-kube-api-access-xz5xc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:29.456247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:29.456243    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b49b552-73e1-413d-a84c-b234f96289b9-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:30.184802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:30.183286    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9" event={"ID":"8b49b552-73e1-413d-a84c-b234f96289b9","Type":"ContainerDied","Data":"4f5363eaa8b3b053c680301831c4f9226d0533f83c7ff4f5b6aadee587a36539"}
Oct 29 23:11:30.184802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:30.183369    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f5363eaa8b3b053c680301831c4f9226d0533f83c7ff4f5b6aadee587a36539"
Oct 29 23:11:30.184802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:30.183447    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"
Oct 29 23:11:31.700199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:31.700164    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"]
Oct 29 23:11:31.700628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:31.700450    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" containerID="cri-o://54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c" gracePeriod=30
Oct 29 23:11:31.714321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:31.714292    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"]
Oct 29 23:11:31.714574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:31.714547    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy" containerID="cri-o://a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2" gracePeriod=30
Oct 29 23:11:32.281678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.281642    4827 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:11:32.281921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.281703    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:11:32.821093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.821060    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"]
Oct 29 23:11:32.822182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.822124    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8b49b552-73e1-413d-a84c-b234f96289b9" containerName="revoke-keys"
Oct 29 23:11:32.822334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.822182    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b49b552-73e1-413d-a84c-b234f96289b9" containerName="revoke-keys"
Oct 29 23:11:32.822414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.822399    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b49b552-73e1-413d-a84c-b234f96289b9" containerName="revoke-keys"
Oct 29 23:11:32.826629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.826608    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.828609 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:32.828571    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:11:32.850944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.850914    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"]
Oct 29 23:11:32.853955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.853892    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-be0401-openshift-default-dockercfg-7qqz7\""
Oct 29 23:11:32.878497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.878469    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-socket\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.878642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.878504    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-data\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.878794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.878751    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-credential-socket\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.878846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.878813    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-podinfo\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.878929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.878914    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhcd\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-kube-api-access-xxhcd\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.878971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.878957    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-token\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.879015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.878986    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istiod-ca-cert\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.879056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.879047    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-envoy\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.879096 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.879077    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-certs\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:32.986110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:32.986065    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhcd\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-kube-api-access-xxhcd\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.011118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.011068    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-token\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.011404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.011376    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istiod-ca-cert\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.011603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.011581    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-envoy\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.011788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.011754    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-certs\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.012246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.012227    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-socket\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.012416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.012397    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-data\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.012757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.012740    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-credential-socket\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.012914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.012897    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-podinfo\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.029795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.015652    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-data\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.029795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.015936    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-socket\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.029795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.016257    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-credential-socket\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.029795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.016607    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istiod-ca-cert\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.029795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.016651    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-certs\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.045266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.045233    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-envoy\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.057576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.057526    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhcd\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-kube-api-access-xxhcd\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.090797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.090703    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-token\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.095791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.093299    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-podinfo\") pod \"e2e-explicit-be0401-openshift-default-8f756b549-xvnjs\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") " pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.162480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.162447    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:33.771597 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:11:33.771559    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e1ead8_0d8b_4e29_9b8a_8d2692ecb3fc.slice/crio-71cec9968d8f1edfd9485f01c2170f29a2563e5c4a4657b5d78ea7eea4c46ed2 WatchSource:0}: Error finding container 71cec9968d8f1edfd9485f01c2170f29a2563e5c4a4657b5d78ea7eea4c46ed2: Status 404 returned error can't find the container with id 71cec9968d8f1edfd9485f01c2170f29a2563e5c4a4657b5d78ea7eea4c46ed2
Oct 29 23:11:33.772019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.771996    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"]
Oct 29 23:11:33.774445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.774411    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:33.774539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.774479    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:33.774539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:33.774510    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:34.274062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:34.274032    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" event={"ID":"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc","Type":"ContainerStarted","Data":"a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b"}
Oct 29 23:11:34.274437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:34.274085    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" event={"ID":"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc","Type":"ContainerStarted","Data":"71cec9968d8f1edfd9485f01c2170f29a2563e5c4a4657b5d78ea7eea4c46ed2"}
Oct 29 23:11:34.333322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:34.333271    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podStartSLOduration=2.333254513 podStartE2EDuration="2.333254513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:34.33042466 +0000 UTC m=+1729.953335679" watchObservedRunningTime="2026-10-29 23:11:34.333254513 +0000 UTC m=+1729.956165532"
Oct 29 23:11:35.166597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:35.166562    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:35.172831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:35.172805    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:35.290614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:35.290581    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:35.291860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:35.291817    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:11:36.373109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.373076    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"]
Oct 29 23:11:36.378006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.377987    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.397684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.397660    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"]
Oct 29 23:11:36.403966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.403943    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:11:36.411860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.404404    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:36.503200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.503167    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b44739f-afba-477f-9533-6f9540de648a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.503387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.503222    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrps\" (UniqueName: \"kubernetes.io/projected/1b44739f-afba-477f-9533-6f9540de648a-kube-api-access-dqrps\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.503560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.503539    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b44739f-afba-477f-9533-6f9540de648a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.605749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.605709    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b44739f-afba-477f-9533-6f9540de648a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.605974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.605931    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b44739f-afba-477f-9533-6f9540de648a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.606064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.605972    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrps\" (UniqueName: \"kubernetes.io/projected/1b44739f-afba-477f-9533-6f9540de648a-kube-api-access-dqrps\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.606573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.606541    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b44739f-afba-477f-9533-6f9540de648a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.608692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.608669    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b44739f-afba-477f-9533-6f9540de648a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.629271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.629213    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrps\" (UniqueName: \"kubernetes.io/projected/1b44739f-afba-477f-9533-6f9540de648a-kube-api-access-dqrps\") pod \"maas-api-revoke-sub-19115b36-zjxf5\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:36.695633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:36.695602    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:37.032283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.032248    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"]
Oct 29 23:11:37.038706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.038678    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:11:37.113933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.113903    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:11:37.220139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220058    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istiod-ca-cert\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.220139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220128    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-data\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.220362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220162    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-podinfo\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.220362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220204    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-envoy\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.220362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220297    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-kube-api-access-tpslq\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-kube-api-access-tpslq\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.220362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220345    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-socket\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.220602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220392    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-kube-api-access-ssvlc\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-kube-api-access-ssvlc\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.220602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220434    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-certs\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.220602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220480    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istiod-ca-cert" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:37.220602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220509    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-credential-socket\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.220602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220554    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-credential-socket\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.220602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220582    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-token\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-token\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.220953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220647    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-data\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.220953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220675    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-socket" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.220953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220785    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-certs\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.220953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220794    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-credential-socket" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.220953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220910    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-data" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.220953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220919    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-token\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-token\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.220953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220956    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-envoy\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.221341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.220982    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istiod-ca-cert\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.221341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.221061    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-socket\") pod \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\" (UID: \"d8ef02ac-a9a8-450b-8df2-739a526ca9e8\") "
Oct 29 23:11:37.221341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.221092    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-podinfo\") pod \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\" (UID: \"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c\") "
Oct 29 23:11:37.221341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.221133    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-data" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.221341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.221285    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-credential-socket" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.221751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.221728    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-certs" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.221916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.221895    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istiod-ca-cert" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:37.221916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.221892    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-socket" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.222111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222087    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222117    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222131    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222144    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222157    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222166    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-certs" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.222189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222170    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222208    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222225    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.222527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.222239    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.320947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.271951    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-podinfo" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:11:37.320947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.286953    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-kube-api-access-ssvlc" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "kube-api-access-ssvlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:37.320947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.302885    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-envoy" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.320947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.320607    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-podinfo" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:11:37.320947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.320697    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-envoy" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:37.320947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.320825    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-token" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:37.320947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.320883    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-kube-api-access-tpslq" pod "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" (UID: "dbd3d2dd-ba9c-426b-a1f1-620be714ba1c"). InnerVolumeSpecName "kube-api-access-tpslq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325301    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325333    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325348    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325361    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325378    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325393    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpslq\" (UniqueName: \"kubernetes.io/projected/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c-kube-api-access-tpslq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325409    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvlc\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-kube-api-access-ssvlc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.346786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.325433    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.370889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.359949    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-token" pod "d8ef02ac-a9a8-450b-8df2-739a526ca9e8" (UID: "d8ef02ac-a9a8-450b-8df2-739a526ca9e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:37.389803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.389231    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5" event={"ID":"1b44739f-afba-477f-9533-6f9540de648a","Type":"ContainerStarted","Data":"5b500e4d085d4a36ef6f078ffdc4d67f6700cea8f0c61ed63a20915eec190217"}
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.399888    4827 generic.go:356] "Generic (PLEG): container finished" podID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerID="54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c" exitCode=0
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.400030    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" event={"ID":"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c","Type":"ContainerDied","Data":"54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c"}
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.400098    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688" event={"ID":"dbd3d2dd-ba9c-426b-a1f1-620be714ba1c","Type":"ContainerDied","Data":"f5bc8db586a0486bc4e5054c56b12192b023fa62d0f1fe7048b412d5e0c3ada4"}
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.400131    4827 scope.go:122] "RemoveContainer" containerID="54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c"
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.400352    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.422348    4827 generic.go:356] "Generic (PLEG): container finished" podID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerID="a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2" exitCode=0
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.423416    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.439727    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" event={"ID":"d8ef02ac-a9a8-450b-8df2-739a526ca9e8","Type":"ContainerDied","Data":"a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2"}
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.439811    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw" event={"ID":"d8ef02ac-a9a8-450b-8df2-739a526ca9e8","Type":"ContainerDied","Data":"950fb5ddc2a194629dd1230087ce1096b97cd2c45a71d448f5d1eb564182f4d4"}
Oct 29 23:11:37.473153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.441491    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ef02ac-a9a8-450b-8df2-739a526ca9e8-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:37.730301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.729685    4827 scope.go:122] "RemoveContainer" containerID="7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90"
Oct 29 23:11:37.745573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.745555    4827 scope.go:122] "RemoveContainer" containerID="54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c"
Oct 29 23:11:37.745933 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:37.745900    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c\": container with ID starting with 54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c not found: ID does not exist" containerID="54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c"
Oct 29 23:11:37.746037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.745944    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c"} err="failed to get container status \"54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c\": rpc error: code = NotFound desc = could not find container \"54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c\": container with ID starting with 54c0421da4e8db708012274b4577ced32863721ae48cfd91d3c17f83e23d734c not found: ID does not exist"
Oct 29 23:11:37.746037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.745968    4827 scope.go:122] "RemoveContainer" containerID="7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90"
Oct 29 23:11:37.746278 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:37.746253    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90\": container with ID starting with 7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90 not found: ID does not exist" containerID="7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90"
Oct 29 23:11:37.746354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.746289    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90"} err="failed to get container status \"7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90\": rpc error: code = NotFound desc = could not find container \"7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90\": container with ID starting with 7f19aba640942d74de55bee5deb4a3cd2ae894921da41a05e45bb6b3f7af9e90 not found: ID does not exist"
Oct 29 23:11:37.746354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.746312    4827 scope.go:122] "RemoveContainer" containerID="a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2"
Oct 29 23:11:37.800788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.784845    4827 scope.go:122] "RemoveContainer" containerID="a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2"
Oct 29 23:11:37.817789 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:37.805181    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2\": container with ID starting with a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2 not found: ID does not exist" containerID="a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2"
Oct 29 23:11:37.817789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.805226    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2"} err="failed to get container status \"a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2\": rpc error: code = NotFound desc = could not find container \"a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2\": container with ID starting with a27ff94cbeb39eb0b1deb7721a6c79d3700c9a3600a907eef1455ee727ff63f2 not found: ID does not exist"
Oct 29 23:11:37.868293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.867863    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"]
Oct 29 23:11:37.917791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.914211    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-j5688"]
Oct 29 23:11:37.990835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:37.989862    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"]
Oct 29 23:11:38.031847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:38.025185    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-6tlkw"]
Oct 29 23:11:38.635102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:38.635066    4827 generic.go:356] "Generic (PLEG): container finished" podID="1b44739f-afba-477f-9533-6f9540de648a" containerID="2e0999d04060088260d053debb069595cc05e2f0d686eae5b9ea741a1bd6b09c" exitCode=0
Oct 29 23:11:38.760634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:38.758160    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" path="/var/lib/kubelet/pods/d8ef02ac-a9a8-450b-8df2-739a526ca9e8/volumes"
Oct 29 23:11:38.760634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:38.759065    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" path="/var/lib/kubelet/pods/dbd3d2dd-ba9c-426b-a1f1-620be714ba1c/volumes"
Oct 29 23:11:38.760634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:38.759642    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5" event={"ID":"1b44739f-afba-477f-9533-6f9540de648a","Type":"ContainerDied","Data":"2e0999d04060088260d053debb069595cc05e2f0d686eae5b9ea741a1bd6b09c"}
Oct 29 23:11:40.561332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.561305    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:40.651158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.651127    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b44739f-afba-477f-9533-6f9540de648a-kube-api-access-dqrps\" (UniqueName: \"kubernetes.io/projected/1b44739f-afba-477f-9533-6f9540de648a-kube-api-access-dqrps\") pod \"1b44739f-afba-477f-9533-6f9540de648a\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") "
Oct 29 23:11:40.651322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.651212    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b44739f-afba-477f-9533-6f9540de648a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b44739f-afba-477f-9533-6f9540de648a-maas-api-openshift-ca\") pod \"1b44739f-afba-477f-9533-6f9540de648a\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") "
Oct 29 23:11:40.651409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.651396    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b44739f-afba-477f-9533-6f9540de648a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b44739f-afba-477f-9533-6f9540de648a-maas-api-xks-ca\") pod \"1b44739f-afba-477f-9533-6f9540de648a\" (UID: \"1b44739f-afba-477f-9533-6f9540de648a\") "
Oct 29 23:11:40.651653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.651626    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44739f-afba-477f-9533-6f9540de648a-maas-api-openshift-ca" pod "1b44739f-afba-477f-9533-6f9540de648a" (UID: "1b44739f-afba-477f-9533-6f9540de648a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:40.652048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.652030    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b44739f-afba-477f-9533-6f9540de648a-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:40.653676 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.653648    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b44739f-afba-477f-9533-6f9540de648a-maas-api-xks-ca" pod "1b44739f-afba-477f-9533-6f9540de648a" (UID: "1b44739f-afba-477f-9533-6f9540de648a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:40.653841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.653823    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b44739f-afba-477f-9533-6f9540de648a-kube-api-access-dqrps" pod "1b44739f-afba-477f-9533-6f9540de648a" (UID: "1b44739f-afba-477f-9533-6f9540de648a"). InnerVolumeSpecName "kube-api-access-dqrps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:40.740365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.740339    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"
Oct 29 23:11:40.740595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.740351    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5" event={"ID":"1b44739f-afba-477f-9533-6f9540de648a","Type":"ContainerDied","Data":"5b500e4d085d4a36ef6f078ffdc4d67f6700cea8f0c61ed63a20915eec190217"}
Oct 29 23:11:40.740654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.740616    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b500e4d085d4a36ef6f078ffdc4d67f6700cea8f0c61ed63a20915eec190217"
Oct 29 23:11:40.754694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.754635    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrps\" (UniqueName: \"kubernetes.io/projected/1b44739f-afba-477f-9533-6f9540de648a-kube-api-access-dqrps\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:40.754694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:40.754658    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b44739f-afba-477f-9533-6f9540de648a-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:11:47.485947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:47.485908    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:11:47.486507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:47.485987    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:51.090782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.090735    4827 generic.go:356] "Generic (PLEG): container finished" podID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerID="722f3586bcc8062d906c98c8c71c5f4ec14d332cbdc8669e7781ff552f5021ca" exitCode=0
Oct 29 23:11:51.091289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.090815    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" event={"ID":"8ea8da08-ef56-4327-becb-86c5dff41b70","Type":"ContainerDied","Data":"722f3586bcc8062d906c98c8c71c5f4ec14d332cbdc8669e7781ff552f5021ca"}
Oct 29 23:11:51.091374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.091295    4827 scope.go:122] "RemoveContainer" containerID="722f3586bcc8062d906c98c8c71c5f4ec14d332cbdc8669e7781ff552f5021ca"
Oct 29 23:11:51.092608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.092588    4827 generic.go:356] "Generic (PLEG): container finished" podID="c2d52c78-f846-4674-9556-28940d3e6c13" containerID="cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069" exitCode=0
Oct 29 23:11:51.092715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.092657    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" event={"ID":"c2d52c78-f846-4674-9556-28940d3e6c13","Type":"ContainerDied","Data":"cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069"}
Oct 29 23:11:51.093056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.093042    4827 scope.go:122] "RemoveContainer" containerID="cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069"
Oct 29 23:11:51.093102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.093080    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:51.093161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.093149    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:51.093201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.093183    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:51.094555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.094528    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:51.094643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.094593    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:51.094643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.094626    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.311105    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"]
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312215    4827 cpu_manager.go:414] "containerMap: removing container" podUID="1b44739f-afba-477f-9533-6f9540de648a" containerName="revoke-keys"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312236    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b44739f-afba-477f-9533-6f9540de648a" containerName="revoke-keys"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312259    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312268    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312283    4827 cpu_manager.go:414] "containerMap: removing container" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312293    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312347    4827 cpu_manager.go:414] "containerMap: removing container" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312355    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312548    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312570    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b44739f-afba-477f-9533-6f9540de648a" containerName="revoke-keys"
Oct 29 23:11:51.314830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.312580    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ef02ac-a9a8-450b-8df2-739a526ca9e8" containerName="istio-proxy"
Oct 29 23:11:51.339557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.339528    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"]
Oct 29 23:11:51.339866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.339651    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.357939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.357744    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-32717a\"/\"openshift-service-ca.crt\""
Oct 29 23:11:51.361107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.357934    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-32717a\"/\"auth-test-model-32717a-kserve-self-signed-certs\""
Oct 29 23:11:51.361107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.357945    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-32717a\"/\"kube-root-ca.crt\""
Oct 29 23:11:51.361107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.357968    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-32717a\"/\"default-dockercfg-vwkfd\""
Oct 29 23:11:51.535788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.532898    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:11:51.575803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.550907    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-dshm\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.575803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.550961    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318f6e68-89e2-451b-b347-ed085617f894-tls-certs\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.575803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.551082    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-model-cache\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.575803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.551208    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-tmp-dir\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.575803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.551240    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-home\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.575803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.551418    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8p7\" (UniqueName: \"kubernetes.io/projected/318f6e68-89e2-451b-b347-ed085617f894-kube-api-access-rv8p7\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.696189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.692690    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-model-cache\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.706809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.706705    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-model-cache\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.743848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.738032    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-tmp-dir\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.743848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.738095    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-home\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.743848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.738301    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8p7\" (UniqueName: \"kubernetes.io/projected/318f6e68-89e2-451b-b347-ed085617f894-kube-api-access-rv8p7\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.743848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.738622    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-dshm\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.743848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.738660    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318f6e68-89e2-451b-b347-ed085617f894-tls-certs\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.743848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.740009    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-tmp-dir\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.743848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.740307    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-home\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.757366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.757247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-dshm\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.839824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.838364    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318f6e68-89e2-451b-b347-ed085617f894-tls-certs\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.839824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.839030    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8p7\" (UniqueName: \"kubernetes.io/projected/318f6e68-89e2-451b-b347-ed085617f894-kube-api-access-rv8p7\") pod \"auth-test-model-32717a-kserve-69bddd4797-6lwsd\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") " pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:51.973205 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:51.973129    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:52.115175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.115129    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" event={"ID":"8ea8da08-ef56-4327-becb-86c5dff41b70","Type":"ContainerStarted","Data":"8097a09fcc6ef7c0dadf35162c4f1855ae5f67f770693baf857297ba2d9ce7a6"}
Oct 29 23:11:52.118834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.118798    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" event={"ID":"c2d52c78-f846-4674-9556-28940d3e6c13","Type":"ContainerStarted","Data":"b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac"}
Oct 29 23:11:52.186176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.186144    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:11:52.186689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.186634    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 29 23:11:52.186996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.186937    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 29 23:11:52.310171 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:11:52.310139    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod318f6e68_89e2_451b_b347_ed085617f894.slice/crio-85a9c8351d8c506ade480f0aad4da94c0d4472b1d6cfd54621d03841cf161adc WatchSource:0}: Error finding container 85a9c8351d8c506ade480f0aad4da94c0d4472b1d6cfd54621d03841cf161adc: Status 404 returned error can't find the container with id 85a9c8351d8c506ade480f0aad4da94c0d4472b1d6cfd54621d03841cf161adc
Oct 29 23:11:52.311988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.311193    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"]
Oct 29 23:11:52.627978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.627930    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:11:52.628538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.628510    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 29 23:11:52.628703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:52.628572    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 29 23:11:53.128941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.128908    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" event={"ID":"318f6e68-89e2-451b-b347-ed085617f894","Type":"ContainerStarted","Data":"4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c"}
Oct 29 23:11:53.129350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.128953    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" event={"ID":"318f6e68-89e2-451b-b347-ed085617f894","Type":"ContainerStarted","Data":"85a9c8351d8c506ade480f0aad4da94c0d4472b1d6cfd54621d03841cf161adc"}
Oct 29 23:11:53.187788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.176985    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podStartSLOduration=2.176963776 podStartE2EDuration="2.176963776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:53.176094383 +0000 UTC m=+1748.799005402" watchObservedRunningTime="2026-10-29 23:11:53.176963776 +0000 UTC m=+1748.799874797"
Oct 29 23:11:53.187788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.182182    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 29 23:11:53.187788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.182263    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 29 23:11:53.628284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.628247    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 29 23:11:53.628496 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.628302    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 29 23:11:53.984843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:53.978869    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:11:54.196260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:54.190193    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 29 23:11:54.196260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:54.190253    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 29 23:11:54.627781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:54.627741    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 29 23:11:54.627968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:54.627813    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 29 23:11:55.198796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:55.197875    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 29 23:11:55.198796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:55.197978    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 29 23:11:55.631898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:55.631860    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 29 23:11:55.632110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:55.631916    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 29 23:11:56.183507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:56.180179    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 29 23:11:56.183507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:56.180239    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 29 23:11:56.628684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:56.628652    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 29 23:11:56.629190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:56.628714    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 29 23:11:57.196245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:57.196208    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 29 23:11:57.196548 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:57.196520    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 29 23:11:57.628150 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:57.628115    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 29 23:11:57.628313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:57.628181    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 29 23:11:58.198790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:58.192311    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 29 23:11:58.198790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:58.192379    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 29 23:11:58.632699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:58.632671    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:11:58.633219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:58.633198    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:11:58.634047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:58.634029    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:11:59.184421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.184391    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:11:59.184761 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.184730    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:11:59.185524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.185507    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:11:59.337819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.337754    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"]
Oct 29 23:11:59.339448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.339419    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd3d2dd-ba9c-426b-a1f1-620be714ba1c" containerName="istio-proxy"
Oct 29 23:11:59.351749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.351717    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.368405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.368377    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-b8f5c4-serving-cert\""
Oct 29 23:11:59.370267 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.370241    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"]
Oct 29 23:11:59.377697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.377671    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.377940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.377781    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsm7d\" (UniqueName: \"kubernetes.io/projected/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-kube-api-access-bsm7d\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.377940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.377919    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.380025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.378195    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.479893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.479803    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.479893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.479871    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsm7d\" (UniqueName: \"kubernetes.io/projected/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-kube-api-access-bsm7d\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.480144 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:59.479896    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-serving-cert: secret "maas-api-e2e-shared-b-w6-b8f5c4-serving-cert" not found
Oct 29 23:11:59.480144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.479940    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.480144 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:11:59.479978    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls podName:9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315 nodeName:}" failed. No retries permitted until 2026-10-29 23:11:59.979954854 +0000 UTC m=+1755.602865863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls") pod "maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9" (UID: "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315") : secret "maas-api-e2e-shared-b-w6-b8f5c4-serving-cert" not found
Oct 29 23:11:59.480144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.480100    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.480583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.480561    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.482515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.482491    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.569897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.568502    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsm7d\" (UniqueName: \"kubernetes.io/projected/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-kube-api-access-bsm7d\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.988475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.988443    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:11:59.991008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:11:59.990980    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:12:00.297969 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:00.296958    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:12:00.728189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:00.725016    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"]
Oct 29 23:12:00.740820 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:12:00.736948    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e10d1eb_2a88_4cbe_a085_9dbf9ef7c315.slice/crio-a480534bbc3bdfb26fef7f31d40b3a969f0c5b5383c2144325b46d0af692d696 WatchSource:0}: Error finding container a480534bbc3bdfb26fef7f31d40b3a969f0c5b5383c2144325b46d0af692d696: Status 404 returned error can't find the container with id a480534bbc3bdfb26fef7f31d40b3a969f0c5b5383c2144325b46d0af692d696
Oct 29 23:12:01.310342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:01.310303    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9" event={"ID":"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315","Type":"ContainerStarted","Data":"61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0"}
Oct 29 23:12:01.310537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:01.310381    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9" event={"ID":"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315","Type":"ContainerStarted","Data":"a480534bbc3bdfb26fef7f31d40b3a969f0c5b5383c2144325b46d0af692d696"}
Oct 29 23:12:01.310587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:01.310542    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:12:01.352675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:01.352622    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9" podStartSLOduration=1.971010187 podStartE2EDuration="2.352606387s" totalImagesPullingTime="381.5962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:01.351762572 +0000 UTC m=+1756.974685857" watchObservedRunningTime="2026-10-29 23:12:01.352606387 +0000 UTC m=+1756.975517406"
Oct 29 23:12:01.973777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:01.973728    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:12:01.986419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:01.986394    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:12:02.330012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:02.329936    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:12:02.825532 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:12:02.825499    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:12:04.325009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:04.323864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": context deadline exceeded"
Oct 29 23:12:04.600621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:04.600524    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:04.863308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:04.862753    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:05.006407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:05.006365    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:05.161791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:05.151864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:05.861905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:05.861862    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:05.868794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:05.863464    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:05.870244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:05.870216    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:12:05.943789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:05.938106    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:12:05.944861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:05.944820    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:12:06.018126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.015839    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.018126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.016373    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:12:06.018126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.017497    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.121958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.120035    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.259206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.259164    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.610350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.610307    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.615610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.615559    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.881447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.881407    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:12:06.883542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.883508    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:12:06.885255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.885225    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:12:06.950743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.941320    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.950743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.941814    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:12:06.959743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.955235    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:06.959743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:06.955651    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:12:07.033207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.031296    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:12:07.041264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.037228    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:07.041264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.037861    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:12:07.052664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.052562    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:12:07.206369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.194177    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:07.206369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.195026    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:12:07.251488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.251062    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:12:07.281992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.281562    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:07.283990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.283053    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:12:07.375694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.367929    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:12:07.405217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.387312    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:12:07.439981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.435835    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:12:07.470984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.468028    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:12:07.505609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.505576    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:12:07.511782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.508275    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:12:07.532686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:07.531335    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:12:12.080179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.080139    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"]
Oct 29 23:12:12.084100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.084074    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.108614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.108586    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"]
Oct 29 23:12:12.110749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.110725    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-be0401-serving-cert\""
Oct 29 23:12:12.170747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.170710    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-metrics-certs\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.170967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.170947    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr765\" (UniqueName: \"kubernetes.io/projected/6a08fd98-9256-4cd1-81e8-b97053154f95-kube-api-access-qr765\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.171066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.171047    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.171153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.171139    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a08fd98-9256-4cd1-81e8-b97053154f95-openshift-service-ca\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.283777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.276067    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a08fd98-9256-4cd1-81e8-b97053154f95-openshift-service-ca\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.283777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.276268    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-metrics-certs\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.283777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.276497    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr765\" (UniqueName: \"kubernetes.io/projected/6a08fd98-9256-4cd1-81e8-b97053154f95-kube-api-access-qr765\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.283777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.276603    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.283777 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:12:12.276902    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-serving-cert: secret "maas-api-e2e-explicit-be0401-serving-cert" not found
Oct 29 23:12:12.283777 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:12:12.276988    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls podName:6a08fd98-9256-4cd1-81e8-b97053154f95 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:12.776965911 +0000 UTC m=+1768.399876923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls") pod "maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb" (UID: "6a08fd98-9256-4cd1-81e8-b97053154f95") : secret "maas-api-e2e-explicit-be0401-serving-cert" not found
Oct 29 23:12:12.303363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.285410    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a08fd98-9256-4cd1-81e8-b97053154f95-openshift-service-ca\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.356010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.346102    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-metrics-certs\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.388505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.388457    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr765\" (UniqueName: \"kubernetes.io/projected/6a08fd98-9256-4cd1-81e8-b97053154f95-kube-api-access-qr765\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.833592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.833555    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:12.836268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:12.836233    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls\") pod \"maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:13.024798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:13.019143    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:13.441839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:13.440544    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"]
Oct 29 23:12:13.765220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:13.765141    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb" event={"ID":"6a08fd98-9256-4cd1-81e8-b97053154f95","Type":"ContainerStarted","Data":"ad620fda6d7ae8b03949bf010827fa548933436daf62554edd11efd00c8fb94b"}
Oct 29 23:12:14.775872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:14.775834    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb" event={"ID":"6a08fd98-9256-4cd1-81e8-b97053154f95","Type":"ContainerStarted","Data":"a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5"}
Oct 29 23:12:14.776269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:14.775923    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:14.819121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:14.819073    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb" podStartSLOduration=2.477697525 podStartE2EDuration="2.819059426s" totalImagesPullingTime="341.361901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:14.818244622 +0000 UTC m=+1770.441155641" watchObservedRunningTime="2026-10-29 23:12:14.819059426 +0000 UTC m=+1770.441970446"
Oct 29 23:12:16.792804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:16.792731    4827 generic.go:356] "Generic (PLEG): container finished" podID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerID="8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db" exitCode=0
Oct 29 23:12:16.793335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:16.792803    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" event={"ID":"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff","Type":"ContainerDied","Data":"8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db"}
Oct 29 23:12:16.793335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:16.793243    4827 scope.go:122] "RemoveContainer" containerID="8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db"
Oct 29 23:12:16.795050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:16.795012    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:16.795167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:16.795087    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:16.795167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:16.795120    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:17.821873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:17.818510    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" event={"ID":"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff","Type":"ContainerStarted","Data":"2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5"}
Oct 29 23:12:18.498395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:18.498303    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:12:18.499085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:18.499005    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 23:12:18.499085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:18.499068    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 23:12:19.498410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:19.498378    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 23:12:19.498934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:19.498429    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 23:12:20.311635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:20.311582    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:12:20.497573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:20.497539    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 23:12:20.497747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:20.497601    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 23:12:20.827789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:20.823153    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:12:21.497693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:21.497646    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 23:12:21.497944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:21.497719    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 23:12:22.520793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:22.498749    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 23:12:22.520793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:22.498828    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 23:12:23.559498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:23.559434    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:12:23.561516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:23.561495    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:12:26.048326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:26.048294    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"]
Oct 29 23:12:26.060917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:26.060889    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f14b1-mzpzc"]
Oct 29 23:12:26.490904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:26.490870    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6438ed73-8266-40ee-b4ca-e55045732844" path="/var/lib/kubelet/pods/6438ed73-8266-40ee-b4ca-e55045732844/volumes"
Oct 29 23:12:32.826951 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:12:32.826913    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:12:40.018173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.018140    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"]
Oct 29 23:12:40.021817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.021798    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.047497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.047465    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"]
Oct 29 23:12:40.053483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.052914    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-c880c7-serving-cert\""
Oct 29 23:12:40.133940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.133910    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qm6\" (UniqueName: \"kubernetes.io/projected/b969954a-8f9f-4a42-9684-2853d025a1fc-kube-api-access-85qm6\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.134130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.134055    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b969954a-8f9f-4a42-9684-2853d025a1fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.134175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.134154    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.134252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.134241    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.236313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.236268    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qm6\" (UniqueName: \"kubernetes.io/projected/b969954a-8f9f-4a42-9684-2853d025a1fc-kube-api-access-85qm6\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.236710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.236687    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b969954a-8f9f-4a42-9684-2853d025a1fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.236955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.236931    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.237188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.237172    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.241237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.241208    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b969954a-8f9f-4a42-9684-2853d025a1fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.309345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.308788    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.344871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.344757    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qm6\" (UniqueName: \"kubernetes.io/projected/b969954a-8f9f-4a42-9684-2853d025a1fc-kube-api-access-85qm6\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.480411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.480376    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.639735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.638874    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:40.788148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:40.788114    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"]
Oct 29 23:12:41.116136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.072212    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"]
Oct 29 23:12:41.116136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.114999    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.195938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.195162    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fba83e-openshift-default-dockercfg-s7cj7\""
Oct 29 23:12:41.214206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.214172    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7kn\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-kube-api-access-mg7kn\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.217714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.217683    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.218077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.218058    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.218226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.218210    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.227640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.227593    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.228066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.228043    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.228385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.228358    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6101a935-9dd1-4075-86ba-087b6c996633-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.231953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.228702    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.232159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.232137    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6101a935-9dd1-4075-86ba-087b6c996633-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.407792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.366966    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.408325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.408296    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.412612 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.412582    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6101a935-9dd1-4075-86ba-087b6c996633-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.663284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.412270    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.663551 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.647449    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6101a935-9dd1-4075-86ba-087b6c996633-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.664129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.664104    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.664285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.664266    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6101a935-9dd1-4075-86ba-087b6c996633-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.664630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.664606    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7kn\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-kube-api-access-mg7kn\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.666825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.666801    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.675384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.675342    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.675661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.675639    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.676552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.676532    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.677065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.677048    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.748789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.682182    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.748789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.736007    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6101a935-9dd1-4075-86ba-087b6c996633-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.748789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.737619    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.754778 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.751596    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.754778 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.752017    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7kn\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-kube-api-access-mg7kn\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.834254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.834218    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:41.839388 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:12:41.839355    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb969954a_8f9f_4a42_9684_2853d025a1fc.slice/crio-7e9cbec4a16a3a7ff957c7d164621d6a84c8c57d46881836433252a835861a23 WatchSource:0}: Error finding container 7e9cbec4a16a3a7ff957c7d164621d6a84c8c57d46881836433252a835861a23: Status 404 returned error can't find the container with id 7e9cbec4a16a3a7ff957c7d164621d6a84c8c57d46881836433252a835861a23
Oct 29 23:12:41.839875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:41.839852    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"]
Oct 29 23:12:42.618294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:42.616565    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f" event={"ID":"b969954a-8f9f-4a42-9684-2853d025a1fc","Type":"ContainerStarted","Data":"7e9cbec4a16a3a7ff957c7d164621d6a84c8c57d46881836433252a835861a23"}
Oct 29 23:12:42.828868 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:12:42.827647    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6101a935_9dd1_4075_86ba_087b6c996633.slice/crio-0251643be2fd509925a355112f974333f6a445ae870a8d56173af9522ed4c5b1 WatchSource:0}: Error finding container 0251643be2fd509925a355112f974333f6a445ae870a8d56173af9522ed4c5b1: Status 404 returned error can't find the container with id 0251643be2fd509925a355112f974333f6a445ae870a8d56173af9522ed4c5b1
Oct 29 23:12:42.829646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:42.829618    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"]
Oct 29 23:12:42.833145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:42.833104    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:42.833240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:42.833191    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:42.833240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:42.833231    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:43.791239 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:43.791197    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f" event={"ID":"b969954a-8f9f-4a42-9684-2853d025a1fc","Type":"ContainerStarted","Data":"0a886643e704824e3957693d140ea1c9f45904eba53a96f264aabd117a3beb81"}
Oct 29 23:12:43.793196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:43.793172    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:43.879144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:43.875176    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" event={"ID":"6101a935-9dd1-4075-86ba-087b6c996633","Type":"ContainerStarted","Data":"0251643be2fd509925a355112f974333f6a445ae870a8d56173af9522ed4c5b1"}
Oct 29 23:12:43.879700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:43.879530    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f" podStartSLOduration=4.461069983 podStartE2EDuration="4.87950927s" totalImagesPullingTime="418.439287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:43.872260006 +0000 UTC m=+1799.495171018" watchObservedRunningTime="2026-10-29 23:12:43.87950927 +0000 UTC m=+1799.502420284"
Oct 29 23:12:43.984922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:43.960739    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podStartSLOduration=3.960716802 podStartE2EDuration="3.960716802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:43.932752911 +0000 UTC m=+1799.555663932" watchObservedRunningTime="2026-10-29 23:12:43.960716802 +0000 UTC m=+1799.583627823"
Oct 29 23:12:44.214788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.207386    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:12:44.437791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.422903    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:12:44.437791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.424547    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.437791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.424876    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:12:44.437791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.425131    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:12:44.490895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.490848    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.513596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.509917    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.581816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.559183    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.675791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.652042    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.839313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.839220    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:44.840122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.840096    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 29 23:12:44.840310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:44.840285    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 29 23:12:45.053792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.046992    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.197792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.147232    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:12:45.287789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.283857    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:12:45.287789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.284805    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:12:45.361793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.345975    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.432058    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.432461    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.432750    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.433029    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.436066    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.436134    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.436182    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:12:45.481619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.436216    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:12:45.531788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.523911    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.531788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.524480    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:12:45.536962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.532913    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.536962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.533316    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:12:45.562127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.562078    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.562946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.562920    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:12:45.670797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.659695    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:12:45.670797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.666303    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 23:12:45.685790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.676258    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:12:45.925912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:45.925879    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:46.057146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.057108    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" event={"ID":"6101a935-9dd1-4075-86ba-087b6c996633","Type":"ContainerStarted","Data":"98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f"}
Oct 29 23:12:46.069060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.069029    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:46.069296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.059638    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:12:46.088947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.082057    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:12:46.121791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.099829    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:12:46.189850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.164212    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.231704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.231662    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:12:46.284056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.284007    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:12:46.351083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.351036    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.363739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.359071    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:12:46.417050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.417021    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:12:46.444623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.444515    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.462881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.457120    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.462881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.457915    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.565630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.565585    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.567161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.565891    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.674890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.674844    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:46.830006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.829928    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:12:46.830266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.830251    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:12:46.830415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.830402    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:12:46.830538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.830527    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:12:46.830675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.830662    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:12:46.831669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.831646    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:12:46.841878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.841857    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:12:46.852818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.852514    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:12:46.860758 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:46.860652    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:12:47.044824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:47.044786    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"]
Oct 29 23:12:47.052755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:47.052729    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a66e9-2cdjj"]
Oct 29 23:12:47.125869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:47.125831    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:12:47.155465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:47.155433    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:12:47.158043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:47.158018    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:12:48.652794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:48.645857    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5363d4-9c46-4624-87a5-d1dae17e1b19" path="/var/lib/kubelet/pods/ce5363d4-9c46-4624-87a5-d1dae17e1b19/volumes"
Oct 29 23:12:49.501182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:49.499940    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:56.066021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.065983    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:12:56.066400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.066042    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 29 23:12:56.420776 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.420732    4827 generic.go:356] "Generic (PLEG): container finished" podID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerID="06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099" exitCode=0
Oct 29 23:12:56.420992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.420806    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" event={"ID":"79663068-a8f3-4945-bcc6-fba2936cd3c5","Type":"ContainerDied","Data":"06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099"}
Oct 29 23:12:56.421333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.421317    4827 scope.go:122] "RemoveContainer" containerID="06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099"
Oct 29 23:12:56.422950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.422925    4827 generic.go:356] "Generic (PLEG): container finished" podID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerID="a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b" exitCode=0
Oct 29 23:12:56.423049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.422984    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" event={"ID":"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc","Type":"ContainerDied","Data":"a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b"}
Oct 29 23:12:56.423362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.423346    4827 scope.go:122] "RemoveContainer" containerID="a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b"
Oct 29 23:12:56.423419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.423391    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:56.423479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.423470    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:56.423543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.423504    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:56.427422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.426115    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:56.427422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.426285    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:56.427422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:56.426388    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:12:57.435022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:57.434953    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" event={"ID":"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc","Type":"ContainerStarted","Data":"2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516"}
Oct 29 23:12:57.437845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:57.437818    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" event={"ID":"79663068-a8f3-4945-bcc6-fba2936cd3c5","Type":"ContainerStarted","Data":"8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6"}
Oct 29 23:12:58.071799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:58.071159    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:12:58.071799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:58.071289    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:12:58.071799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:58.071347    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 29 23:12:58.184794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:58.180267    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:12:58.184794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:58.180397    4827 patch_prober.go:29] interesting pod/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 29 23:12:58.184794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:58.180438    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 29 23:12:59.078856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:59.075450    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:12:59.082863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:59.082826    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 29 23:12:59.189791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:59.189516    4827 patch_prober.go:29] interesting pod/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 29 23:12:59.189791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:12:59.189577    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 29 23:13:00.071802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:00.071191    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:13:00.071802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:00.071249    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 29 23:13:00.180793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:00.167159    4827 patch_prober.go:29] interesting pod/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 29 23:13:00.180793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:00.167230    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 29 23:13:01.078174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:01.078133    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:13:01.078474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:01.078446    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 29 23:13:01.186793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:01.185887    4827 patch_prober.go:29] interesting pod/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 29 23:13:01.186793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:01.185950    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 29 23:13:02.071928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:02.071892    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:13:02.072440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:02.072409    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 29 23:13:02.184791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:02.184631    4827 patch_prober.go:29] interesting pod/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 29 23:13:02.184791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:02.184701    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 29 23:13:03.020590 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:13:03.019980    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:13:03.087975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:03.087934    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:13:03.088187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:03.088000    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 29 23:13:03.170803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:03.170177    4827 patch_prober.go:29] interesting pod/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 29 23:13:03.170803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:03.170236    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 29 23:13:04.075415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:04.075374    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:13:04.075881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:04.075860    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:13:04.076672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:04.076654    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:13:04.171791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:04.171750    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:13:04.172112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:04.172092    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:13:04.172808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:04.172789    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:13:08.073723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.073684    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"]
Oct 29 23:13:08.078519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.078500    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.104551 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.104520    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"]
Oct 29 23:13:08.163507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.163476    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.163732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.163524    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.163732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.163616    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh548\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-kube-api-access-wh548\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.163732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.163680    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.163942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.163794    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.163942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.163864    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.164064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.163974    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.164064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.164033    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.164141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.164127    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.265820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.265751    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.265995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.265851    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.265995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.265886    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh548\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-kube-api-access-wh548\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.265995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.265961    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266021    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266094    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266139    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266179    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266280    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266361    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266556    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266731    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266739    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.266871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.266832    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.268365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.268342    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.268581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.268560    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.292074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.292052    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.292202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.292119    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh548\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-kube-api-access-wh548\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.395156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.395127    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:08.719539 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:13:08.719508    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b00b28_26d5_4247_871e_1fc3bbe8f020.slice/crio-3df909c4a42b3cdc5b7a751deb6d430441965a3b420d55b99dc3711dce9fddea WatchSource:0}: Error finding container 3df909c4a42b3cdc5b7a751deb6d430441965a3b420d55b99dc3711dce9fddea: Status 404 returned error can't find the container with id 3df909c4a42b3cdc5b7a751deb6d430441965a3b420d55b99dc3711dce9fddea
Oct 29 23:13:08.720233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.720207    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"]
Oct 29 23:13:08.722508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.722476    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:08.722572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.722548    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:08.722610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:08.722586    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:09.634704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:09.634672    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" event={"ID":"b3b00b28-26d5-4247-871e-1fc3bbe8f020","Type":"ContainerStarted","Data":"83e0fec097c94f67f7850078e4906476e603aa090327ba83c4cddc9c03b3ce79"}
Oct 29 23:13:09.635129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:09.634734    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" event={"ID":"b3b00b28-26d5-4247-871e-1fc3bbe8f020","Type":"ContainerStarted","Data":"3df909c4a42b3cdc5b7a751deb6d430441965a3b420d55b99dc3711dce9fddea"}
Oct 29 23:13:09.695506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:09.695448    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podStartSLOduration=1.695430486 podStartE2EDuration="1.695430486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:09.689912364 +0000 UTC m=+1825.312823385" watchObservedRunningTime="2026-10-29 23:13:09.695430486 +0000 UTC m=+1825.318341506"
Oct 29 23:13:10.396157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:10.396123    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:10.398251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:10.398222    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 29 23:13:10.398445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:10.398271    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 29 23:13:11.396295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:11.396262    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 29 23:13:11.396712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:11.396338    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 29 23:13:12.396217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:12.396184    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 29 23:13:12.396465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:12.396439    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 29 23:13:13.396347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:13.396312    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 29 23:13:13.396545 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:13.396377    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 29 23:13:14.401352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.401316    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 29 23:13:14.401999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.401963    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 29 23:13:14.672832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.671877    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:13:14.683820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.683778    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:13:14.696791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.686215    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:13:14.772381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.772340    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:13:14.835220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.834098    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:13:14.851327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:14.851293    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:13:15.397349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:15.397308    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 29 23:13:15.397675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:15.397629    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 29 23:13:16.396417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:16.396382    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 29 23:13:16.396991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:16.396441    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 29 23:13:17.400971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:17.400941    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:17.401473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:17.401449    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:17.402104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:17.402081    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:13:19.031662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:19.031629    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"]
Oct 29 23:13:19.043589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:19.043540    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-415742ef-rzl8x"]
Oct 29 23:13:20.490619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:20.490577    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae11e6d-c37a-4b0e-9de7-a746713a8840" path="/var/lib/kubelet/pods/fae11e6d-c37a-4b0e-9de7-a746713a8840/volumes"
Oct 29 23:13:23.598211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.598175    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"]
Oct 29 23:13:23.604386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.604369    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.631015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.630986    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fba83e-serving-cert\""
Oct 29 23:13:23.637070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.637042    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"]
Oct 29 23:13:23.691728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.691695    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a99063-d0e7-4e39-93c8-b210080df169-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.692165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.692137    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-metrics-certs\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.692240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.692197    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.692343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.692324    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hlp\" (UniqueName: \"kubernetes.io/projected/d9a99063-d0e7-4e39-93c8-b210080df169-kube-api-access-l5hlp\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.800949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.800905    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-metrics-certs\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.801724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.801701    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.802120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.802086    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hlp\" (UniqueName: \"kubernetes.io/projected/d9a99063-d0e7-4e39-93c8-b210080df169-kube-api-access-l5hlp\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.809045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.809003    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a99063-d0e7-4e39-93c8-b210080df169-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.824592 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:13:23.801973    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-serving-cert: secret "maas-api-e2e-inf-a-fba83e-serving-cert" not found
Oct 29 23:13:23.825716 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:13:23.825695    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls podName:d9a99063-d0e7-4e39-93c8-b210080df169 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:24.325666657 +0000 UTC m=+1839.948577678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls") pod "maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6" (UID: "d9a99063-d0e7-4e39-93c8-b210080df169") : secret "maas-api-e2e-inf-a-fba83e-serving-cert" not found
Oct 29 23:13:23.826123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.810345    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a99063-d0e7-4e39-93c8-b210080df169-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.900880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.900847    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-metrics-certs\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:23.920408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:23.920322    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hlp\" (UniqueName: \"kubernetes.io/projected/d9a99063-d0e7-4e39-93c8-b210080df169-kube-api-access-l5hlp\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:24.425824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:24.425787    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:24.428450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:24.428419    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls\") pod \"maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:24.538159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:24.538118    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:24.992524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:24.992489    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"]
Oct 29 23:13:25.015803 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:13:24.999291    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a99063_d0e7_4e39_93c8_b210080df169.slice/crio-61b5285f069685f2701e5e8d3bdf6b6c1076decfb5d5102ee6473b87996fa2e1 WatchSource:0}: Error finding container 61b5285f069685f2701e5e8d3bdf6b6c1076decfb5d5102ee6473b87996fa2e1: Status 404 returned error can't find the container with id 61b5285f069685f2701e5e8d3bdf6b6c1076decfb5d5102ee6473b87996fa2e1
Oct 29 23:13:26.010844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.010808    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6" event={"ID":"d9a99063-d0e7-4e39-93c8-b210080df169","Type":"ContainerStarted","Data":"3c0010100a7d8cd5b5b46f369f3a7e1f912d4a9980ad238b9cec1451976b24ea"}
Oct 29 23:13:26.011269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.010869    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6" event={"ID":"d9a99063-d0e7-4e39-93c8-b210080df169","Type":"ContainerStarted","Data":"61b5285f069685f2701e5e8d3bdf6b6c1076decfb5d5102ee6473b87996fa2e1"}
Oct 29 23:13:26.011269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.010914    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:26.056103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.056048    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6" podStartSLOduration=2.549420022 podStartE2EDuration="3.056030778s" totalImagesPullingTime="506.610756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:26.052843349 +0000 UTC m=+1841.675754369" watchObservedRunningTime="2026-10-29 23:13:26.056030778 +0000 UTC m=+1841.678941798"
Oct 29 23:13:26.619700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.619668    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"]
Oct 29 23:13:26.624474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.624452    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.644820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.644795    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"]
Oct 29 23:13:26.651016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.650989    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-5e52e0-openshift-default-dockercfg-w8nms\""
Oct 29 23:13:26.802797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.790106    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-data\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.818883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.813416    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-token\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.818883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.813462    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istiod-ca-cert\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.818883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.813638    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-podinfo\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.818883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.813670    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7j2s\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-kube-api-access-l7j2s\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.818883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.813733    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-certs\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.905184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.905140    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-envoy\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.905388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.905200    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-socket\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:26.905462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:26.905421    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-credential-socket\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.065263    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-podinfo\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.065318    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7j2s\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-kube-api-access-l7j2s\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.065394    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-certs\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.065522    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-envoy\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.065563    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-socket\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.065801    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-credential-socket\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.065948    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-data\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.066028    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-token\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.066056    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istiod-ca-cert\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.066989    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istiod-ca-cert\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.067316    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-socket\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.085866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.067553    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-credential-socket\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.132299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.131277    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-data\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.194549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.191248    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-certs\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.196271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.196234    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-podinfo\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.206534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.206500    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-envoy\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.284844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.267352    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-token\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.317387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.317347    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7j2s\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-kube-api-access-l7j2s\") pod \"e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") " pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.539751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.539675    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:27.865656 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:13:27.865626    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38d50dd_ecfc_48ec_9dee_ea03ac789908.slice/crio-b49d611ba217557b11098f5fb934ca5a1f419e3a085899cebe2657cfe9ccf67c WatchSource:0}: Error finding container b49d611ba217557b11098f5fb934ca5a1f419e3a085899cebe2657cfe9ccf67c: Status 404 returned error can't find the container with id b49d611ba217557b11098f5fb934ca5a1f419e3a085899cebe2657cfe9ccf67c
Oct 29 23:13:27.867555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.867528    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"]
Oct 29 23:13:27.869418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.869382    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:27.869514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.869456    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:27.869514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:27.869493    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:28.332513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:28.332479    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" event={"ID":"c38d50dd-ecfc-48ec-9dee-ea03ac789908","Type":"ContainerStarted","Data":"50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417"}
Oct 29 23:13:28.332959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:28.332535    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" event={"ID":"c38d50dd-ecfc-48ec-9dee-ea03ac789908","Type":"ContainerStarted","Data":"b49d611ba217557b11098f5fb934ca5a1f419e3a085899cebe2657cfe9ccf67c"}
Oct 29 23:13:28.393414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:28.393364    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podStartSLOduration=2.393349906 podStartE2EDuration="2.393349906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:28.390508042 +0000 UTC m=+1844.013419062" watchObservedRunningTime="2026-10-29 23:13:28.393349906 +0000 UTC m=+1844.016260925"
Oct 29 23:13:28.543791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:28.542182    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:28.620514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:28.620424    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:29.341351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:29.341323    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:29.342294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:29.342270    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:13:32.327187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:32.327158    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:13:33.036198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:33.036163    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"]
Oct 29 23:13:33.044885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:33.043389    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1c7cef5-5scvd"]
Oct 29 23:13:33.068232 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:13:33.068195    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:13:34.491479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:34.491450    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc5d5f2-53e9-4ef2-8a04-494293167c6b" path="/var/lib/kubelet/pods/8dc5d5f2-53e9-4ef2-8a04-494293167c6b/volumes"
Oct 29 23:13:35.031173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:35.031135    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"]
Oct 29 23:13:35.042829 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:35.042800    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76da0b-n6tls"]
Oct 29 23:13:36.490274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:36.490240    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be891873-b788-49ad-b545-a3fa4842851b" path="/var/lib/kubelet/pods/be891873-b788-49ad-b545-a3fa4842851b/volumes"
Oct 29 23:13:37.469662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.469622    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"]
Oct 29 23:13:37.484220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.484185    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.501328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.501296    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"]
Oct 29 23:13:37.501724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.501557    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-be0401\"/\"explicit-model-be0401-kserve-self-signed-certs\""
Oct 29 23:13:37.501724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.501656    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-be0401\"/\"kube-root-ca.crt\""
Oct 29 23:13:37.501724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.501717    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-be0401\"/\"openshift-service-ca.crt\""
Oct 29 23:13:37.501908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.501782    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-be0401\"/\"default-dockercfg-kp8z7\""
Oct 29 23:13:37.611701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.611669    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-dshm\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.611862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.611848    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-tmp-dir\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.611923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.611888    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d702e12-6a87-46d7-a920-19c0908a530c-tls-certs\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.612053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.612030    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-model-cache\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.612092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.612080    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgpct\" (UniqueName: \"kubernetes.io/projected/4d702e12-6a87-46d7-a920-19c0908a530c-kube-api-access-vgpct\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.612136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.612126    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-home\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.742845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.739582    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-dshm\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.764049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.764010    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-tmp-dir\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.764358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.764335    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d702e12-6a87-46d7-a920-19c0908a530c-tls-certs\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.764589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.764575    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-model-cache\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.764747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.764727    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpct\" (UniqueName: \"kubernetes.io/projected/4d702e12-6a87-46d7-a920-19c0908a530c-kube-api-access-vgpct\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.764920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.764904    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-home\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.766464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.766441    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-tmp-dir\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.786285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.768486    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-model-cache\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.786285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.768728    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-home\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.787174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.787144    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d702e12-6a87-46d7-a920-19c0908a530c-tls-certs\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.815788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.808444    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-dshm\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:37.915828 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:37.882341    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpct\" (UniqueName: \"kubernetes.io/projected/4d702e12-6a87-46d7-a920-19c0908a530c-kube-api-access-vgpct\") pod \"explicit-model-be0401-kserve-78898fb67f-l2zn7\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") " pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:38.105252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:38.105157    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:38.415907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:38.415876    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"]
Oct 29 23:13:38.416079 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:13:38.416061    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d702e12_6a87_46d7_a920_19c0908a530c.slice/crio-451616c9ca13b279aec2efc266fb1e04a8d604bc6201a5fb2976959120f6b924 WatchSource:0}: Error finding container 451616c9ca13b279aec2efc266fb1e04a8d604bc6201a5fb2976959120f6b924: Status 404 returned error can't find the container with id 451616c9ca13b279aec2efc266fb1e04a8d604bc6201a5fb2976959120f6b924
Oct 29 23:13:38.511788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:38.511732    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7" event={"ID":"4d702e12-6a87-46d7-a920-19c0908a530c","Type":"ContainerStarted","Data":"d9f07599369f39673020727ea161062d065b69a075acec461dd26a8ec3282455"}
Oct 29 23:13:38.512249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:38.511806    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7" event={"ID":"4d702e12-6a87-46d7-a920-19c0908a530c","Type":"ContainerStarted","Data":"451616c9ca13b279aec2efc266fb1e04a8d604bc6201a5fb2976959120f6b924"}
Oct 29 23:13:38.558550 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:38.558490    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7" podStartSLOduration=1.558475901 podStartE2EDuration="1.558475901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:38.553222353 +0000 UTC m=+1854.176133372" watchObservedRunningTime="2026-10-29 23:13:38.558475901 +0000 UTC m=+1854.181386921"
Oct 29 23:13:39.105706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.105674    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:39.276955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.276918    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"]
Oct 29 23:13:39.283084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.283059    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.300753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.300729    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-67363b-openshift-default-dockercfg-kr7r9\""
Oct 29 23:13:39.316714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.316684    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"]
Oct 29 23:13:39.333887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.333854    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334009    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55ws\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-kube-api-access-n55ws\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334158    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334218    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de570536-b5be-4262-b005-499d5113ae21-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334382    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de570536-b5be-4262-b005-499d5113ae21-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334484    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334544    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334676 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334660    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.334799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.334754    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436026    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436351    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436540    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436589    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55ws\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-kube-api-access-n55ws\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436541    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436711    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436781    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de570536-b5be-4262-b005-499d5113ae21-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436822    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436877    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de570536-b5be-4262-b005-499d5113ae21-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.436952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436924    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.437141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.436951    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.437292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.437264    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.437362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.437319    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.437521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.437499    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de570536-b5be-4262-b005-499d5113ae21-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.439960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.439938    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.440189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.440172    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de570536-b5be-4262-b005-499d5113ae21-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.460265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.460241    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.460398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.460242    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55ws\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-kube-api-access-n55ws\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.599530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.599497    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:39.926336 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:13:39.926296    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde570536_b5be_4262_b005_499d5113ae21.slice/crio-7539d033b90fca91ef0951a52325ecf87a8fdd18182b2973c23ab0453612b636 WatchSource:0}: Error finding container 7539d033b90fca91ef0951a52325ecf87a8fdd18182b2973c23ab0453612b636: Status 404 returned error can't find the container with id 7539d033b90fca91ef0951a52325ecf87a8fdd18182b2973c23ab0453612b636
Oct 29 23:13:39.929009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.928982    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"]
Oct 29 23:13:39.929861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.929823    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:39.929948 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.929892    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:39.929948 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:39.929922    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:13:40.532625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:40.532592    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" event={"ID":"de570536-b5be-4262-b005-499d5113ae21","Type":"ContainerStarted","Data":"c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6"}
Oct 29 23:13:40.532882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:40.532648    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" event={"ID":"de570536-b5be-4262-b005-499d5113ae21","Type":"ContainerStarted","Data":"7539d033b90fca91ef0951a52325ecf87a8fdd18182b2973c23ab0453612b636"}
Oct 29 23:13:40.600540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:40.600490    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:40.603093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:40.603026    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podStartSLOduration=1.603006239 podStartE2EDuration="1.603006239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:40.595457222 +0000 UTC m=+1856.218368243" watchObservedRunningTime="2026-10-29 23:13:40.603006239 +0000 UTC m=+1856.225917260"
Oct 29 23:13:41.602235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:41.602199    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 29 23:13:41.602709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:41.602259    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 29 23:13:42.562787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:42.562730    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"]
Oct 29 23:13:42.601345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:42.601309    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 29 23:13:42.601569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:42.601383    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 29 23:13:43.600059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:43.600013    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 29 23:13:43.600646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:43.600091    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 29 23:13:44.600625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:44.600591    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 29 23:13:44.601011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:44.600637    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 29 23:13:45.600145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:45.600102    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 29 23:13:45.600368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:45.600171    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 29 23:13:46.031430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:46.031399    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"]
Oct 29 23:13:46.036421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:46.036391    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f68f16-p7n97"]
Oct 29 23:13:46.495216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:46.495178    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af01c502-f7a1-4b01-8738-8a0a55676506" path="/var/lib/kubelet/pods/af01c502-f7a1-4b01-8738-8a0a55676506/volumes"
Oct 29 23:13:46.600256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:46.600226    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 29 23:13:46.600443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:46.600290    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 29 23:13:47.604903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:47.604875    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:47.605386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:47.605366    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:47.606132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:47.606114    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:13:49.643102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:49.643069    4827 scope.go:122] "RemoveContainer" containerID="18c537f68811d28a0fb8ebee80ce7d1b317ab0b6974702d6e12e6271d270e122"
Oct 29 23:13:49.653564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:49.653546    4827 scope.go:122] "RemoveContainer" containerID="06383392850a123bb01d7f6763974df188025320df8c266b1105e150414a059a"
Oct 29 23:13:49.663912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:49.663887    4827 scope.go:122] "RemoveContainer" containerID="37bca9cbe4e8efb5b301a6b2c604eb6a824661cad373eb1ffea764bb156c2ec0"
Oct 29 23:13:49.674382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:49.674365    4827 scope.go:122] "RemoveContainer" containerID="67301542fbfae44bd6cd7956f07c385d06adc10825844e5f9845fa2f07039f04"
Oct 29 23:13:57.032324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:57.032289    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"]
Oct 29 23:13:57.044002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:57.043975    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8bd2175-vxl64"]
Oct 29 23:13:57.621325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:57.621262    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7" podUID="4d702e12-6a87-46d7-a920-19c0908a530c" containerName="main" containerID="cri-o://d9f07599369f39673020727ea161062d065b69a075acec461dd26a8ec3282455" gracePeriod=45
Oct 29 23:13:57.842046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:57.842010    4827 generic.go:356] "Generic (PLEG): container finished" podID="4d702e12-6a87-46d7-a920-19c0908a530c" containerID="d9f07599369f39673020727ea161062d065b69a075acec461dd26a8ec3282455" exitCode=0
Oct 29 23:13:57.843242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:57.843214    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7" event={"ID":"4d702e12-6a87-46d7-a920-19c0908a530c","Type":"ContainerDied","Data":"d9f07599369f39673020727ea161062d065b69a075acec461dd26a8ec3282455"}
Oct 29 23:13:58.041814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.041784    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:58.189337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189292    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d702e12-6a87-46d7-a920-19c0908a530c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d702e12-6a87-46d7-a920-19c0908a530c-tls-certs\") pod \"4d702e12-6a87-46d7-a920-19c0908a530c\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") "
Oct 29 23:13:58.189514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189425    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-model-cache\") pod \"4d702e12-6a87-46d7-a920-19c0908a530c\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") "
Oct 29 23:13:58.189611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189592    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-home\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-home\") pod \"4d702e12-6a87-46d7-a920-19c0908a530c\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") "
Oct 29 23:13:58.189677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189661    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d702e12-6a87-46d7-a920-19c0908a530c-kube-api-access-vgpct\" (UniqueName: \"kubernetes.io/projected/4d702e12-6a87-46d7-a920-19c0908a530c-kube-api-access-vgpct\") pod \"4d702e12-6a87-46d7-a920-19c0908a530c\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") "
Oct 29 23:13:58.189744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189667    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-model-cache" pod "4d702e12-6a87-46d7-a920-19c0908a530c" (UID: "4d702e12-6a87-46d7-a920-19c0908a530c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:58.189864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189814    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-home" pod "4d702e12-6a87-46d7-a920-19c0908a530c" (UID: "4d702e12-6a87-46d7-a920-19c0908a530c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:58.189936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189889    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-dshm\") pod \"4d702e12-6a87-46d7-a920-19c0908a530c\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") "
Oct 29 23:13:58.189997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.189962    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-tmp-dir\") pod \"4d702e12-6a87-46d7-a920-19c0908a530c\" (UID: \"4d702e12-6a87-46d7-a920-19c0908a530c\") "
Oct 29 23:13:58.190267 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.190241    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-tmp-dir" pod "4d702e12-6a87-46d7-a920-19c0908a530c" (UID: "4d702e12-6a87-46d7-a920-19c0908a530c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:58.190739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.190717    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:13:58.190887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.190741    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:13:58.190887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.190752    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:13:58.191681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.191656    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d702e12-6a87-46d7-a920-19c0908a530c-tls-certs" pod "4d702e12-6a87-46d7-a920-19c0908a530c" (UID: "4d702e12-6a87-46d7-a920-19c0908a530c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:13:58.191884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.191866    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-dshm" pod "4d702e12-6a87-46d7-a920-19c0908a530c" (UID: "4d702e12-6a87-46d7-a920-19c0908a530c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:58.191949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.191892    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d702e12-6a87-46d7-a920-19c0908a530c-kube-api-access-vgpct" pod "4d702e12-6a87-46d7-a920-19c0908a530c" (UID: "4d702e12-6a87-46d7-a920-19c0908a530c"). InnerVolumeSpecName "kube-api-access-vgpct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:58.292660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.292568    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d702e12-6a87-46d7-a920-19c0908a530c-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:13:58.292660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.292600    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgpct\" (UniqueName: \"kubernetes.io/projected/4d702e12-6a87-46d7-a920-19c0908a530c-kube-api-access-vgpct\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:13:58.292660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.292612    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d702e12-6a87-46d7-a920-19c0908a530c-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:13:58.490505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.490472    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f788d17a-65b2-4bb4-bbc9-53c1dd4dea02" path="/var/lib/kubelet/pods/f788d17a-65b2-4bb4-bbc9-53c1dd4dea02/volumes"
Oct 29 23:13:58.853554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.853528    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"
Oct 29 23:13:58.854308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.853523    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7" event={"ID":"4d702e12-6a87-46d7-a920-19c0908a530c","Type":"ContainerDied","Data":"451616c9ca13b279aec2efc266fb1e04a8d604bc6201a5fb2976959120f6b924"}
Oct 29 23:13:58.854308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.853653    4827 scope.go:122] "RemoveContainer" containerID="d9f07599369f39673020727ea161062d065b69a075acec461dd26a8ec3282455"
Oct 29 23:13:58.910862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.910828    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"]
Oct 29 23:13:58.920585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:58.920551    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-be0401/explicit-model-be0401-kserve-78898fb67f-l2zn7"]
Oct 29 23:13:59.039865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:59.039820    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"]
Oct 29 23:13:59.044467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:13:59.044435    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0612f85-gqhrl"]
Oct 29 23:14:00.490196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:00.490165    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d702e12-6a87-46d7-a920-19c0908a530c" path="/var/lib/kubelet/pods/4d702e12-6a87-46d7-a920-19c0908a530c/volumes"
Oct 29 23:14:00.490716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:00.490700    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ddd219-2515-4f10-92fc-cf7398d766bc" path="/var/lib/kubelet/pods/f4ddd219-2515-4f10-92fc-cf7398d766bc/volumes"
Oct 29 23:14:03.176290 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:14:03.176059    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:14:03.664584 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:03.664521    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:03.714801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:03.698695    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:03.774299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:03.774252    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:03.901796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:03.873916    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:03.946094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:03.945995    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:04.276840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.272883    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:04.432890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.432842    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:04.450885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.450845    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:04.503789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.492068    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:04.707114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.703878    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": context deadline exceeded"
Oct 29 23:14:04.707114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.704302    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:14:04.707114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.704977    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:14:04.899788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.888206    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:14:04.899788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.888583    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:14:04.899788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.889170    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:14:04.985634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.985548    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:14:05.022788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:04.997294    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:05.102821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.077552    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:14:05.275844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.274012    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:05.275844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.275096    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:14:05.438790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.409288    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:14:05.690789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.687365    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:05.742795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.740970    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:05.742795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.741605    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": context deadline exceeded"
Oct 29 23:14:05.742795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.741954    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:14:05.770789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.763272    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:14:05.857193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.837393    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:14:05.857193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:05.856133    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:14:06.341716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:06.341675    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"]
Oct 29 23:14:06.341982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:06.341942    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb" podUID="6a08fd98-9256-4cd1-81e8-b97053154f95" containerName="maas-api" containerID="cri-o://a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5" gracePeriod=30
Oct 29 23:14:06.780428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:06.779985    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:14:07.004856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:06.981154    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a08fd98-9256-4cd1-81e8-b97053154f95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a08fd98-9256-4cd1-81e8-b97053154f95-openshift-service-ca\") pod \"6a08fd98-9256-4cd1-81e8-b97053154f95\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") "
Oct 29 23:14:07.026815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.026783    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls\") pod \"6a08fd98-9256-4cd1-81e8-b97053154f95\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") "
Oct 29 23:14:07.027480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.026560    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a08fd98-9256-4cd1-81e8-b97053154f95-openshift-service-ca" pod "6a08fd98-9256-4cd1-81e8-b97053154f95" (UID: "6a08fd98-9256-4cd1-81e8-b97053154f95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:07.027846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.027822    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-metrics-certs\") pod \"6a08fd98-9256-4cd1-81e8-b97053154f95\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") "
Oct 29 23:14:07.039464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.039436    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a08fd98-9256-4cd1-81e8-b97053154f95-kube-api-access-qr765\" (UniqueName: \"kubernetes.io/projected/6a08fd98-9256-4cd1-81e8-b97053154f95-kube-api-access-qr765\") pod \"6a08fd98-9256-4cd1-81e8-b97053154f95\" (UID: \"6a08fd98-9256-4cd1-81e8-b97053154f95\") "
Oct 29 23:14:07.061032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.060998    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a08fd98-9256-4cd1-81e8-b97053154f95-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:07.086011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.085976    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a08fd98-9256-4cd1-81e8-b97053154f95-kube-api-access-qr765" pod "6a08fd98-9256-4cd1-81e8-b97053154f95" (UID: "6a08fd98-9256-4cd1-81e8-b97053154f95"). InnerVolumeSpecName "kube-api-access-qr765". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:07.126957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.126900    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-metrics-certs" pod "6a08fd98-9256-4cd1-81e8-b97053154f95" (UID: "6a08fd98-9256-4cd1-81e8-b97053154f95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:07.160888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.127250    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls" pod "6a08fd98-9256-4cd1-81e8-b97053154f95" (UID: "6a08fd98-9256-4cd1-81e8-b97053154f95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:07.184787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.163575    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:07.187973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.187949    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a08fd98-9256-4cd1-81e8-b97053154f95-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:07.188195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.188180    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr765\" (UniqueName: \"kubernetes.io/projected/6a08fd98-9256-4cd1-81e8-b97053154f95-kube-api-access-qr765\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:07.657601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.657567    4827 generic.go:356] "Generic (PLEG): container finished" podID="6a08fd98-9256-4cd1-81e8-b97053154f95" containerID="a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5" exitCode=0
Oct 29 23:14:07.657848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.657816    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb" event={"ID":"6a08fd98-9256-4cd1-81e8-b97053154f95","Type":"ContainerDied","Data":"a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5"}
Oct 29 23:14:07.657941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.657879    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb" event={"ID":"6a08fd98-9256-4cd1-81e8-b97053154f95","Type":"ContainerDied","Data":"ad620fda6d7ae8b03949bf010827fa548933436daf62554edd11efd00c8fb94b"}
Oct 29 23:14:07.657941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.657913    4827 scope.go:122] "RemoveContainer" containerID="a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5"
Oct 29 23:14:07.658140 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.658111    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"
Oct 29 23:14:07.699635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.699608    4827 scope.go:122] "RemoveContainer" containerID="a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5"
Oct 29 23:14:07.700082 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:14:07.700053    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5\": container with ID starting with a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5 not found: ID does not exist" containerID="a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5"
Oct 29 23:14:07.700168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.700102    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5"} err="failed to get container status \"a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5\": rpc error: code = NotFound desc = could not find container \"a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5\": container with ID starting with a01bfdfca579c0079a0175a3f086e691babd5a8f83f5267e88e02a00ac662df5 not found: ID does not exist"
Oct 29 23:14:07.724530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.724505    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"]
Oct 29 23:14:07.755789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:07.744279    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-be0401-7cfb89dbbc-6nmbb"]
Oct 29 23:14:08.083499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:08.083317    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"]
Oct 29 23:14:08.096840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:08.096806    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f398e-frbl8"]
Oct 29 23:14:08.506719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:08.506688    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63da891d-f1e0-4cdc-b9ce-1b8212f49b01" path="/var/lib/kubelet/pods/63da891d-f1e0-4cdc-b9ce-1b8212f49b01/volumes"
Oct 29 23:14:08.507341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:08.507325    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a08fd98-9256-4cd1-81e8-b97053154f95" path="/var/lib/kubelet/pods/6a08fd98-9256-4cd1-81e8-b97053154f95/volumes"
Oct 29 23:14:14.029711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:14.029675    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"]
Oct 29 23:14:14.041754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:14.041719    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b0fc18-tht9b"]
Oct 29 23:14:14.498965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:14.498932    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8341e77-16de-40ab-be80-82277155c7ee" path="/var/lib/kubelet/pods/f8341e77-16de-40ab-be80-82277155c7ee/volumes"
Oct 29 23:14:15.472131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:15.472097    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"]
Oct 29 23:14:15.472639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:15.472394    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" containerID="cri-o://2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516" gracePeriod=30
Oct 29 23:14:16.745569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.745535    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"]
Oct 29 23:14:16.749537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.749505    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4d702e12-6a87-46d7-a920-19c0908a530c" containerName="main"
Oct 29 23:14:16.749537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.749536    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d702e12-6a87-46d7-a920-19c0908a530c" containerName="main"
Oct 29 23:14:16.749713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.749562    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6a08fd98-9256-4cd1-81e8-b97053154f95" containerName="maas-api"
Oct 29 23:14:16.749713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.749570    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a08fd98-9256-4cd1-81e8-b97053154f95" containerName="maas-api"
Oct 29 23:14:16.749846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.749808    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a08fd98-9256-4cd1-81e8-b97053154f95" containerName="maas-api"
Oct 29 23:14:16.749890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.749846    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d702e12-6a87-46d7-a920-19c0908a530c" containerName="main"
Oct 29 23:14:16.755660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.755631    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795165    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-podinfo\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795254    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-credential-socket\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795296    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-certs\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795357    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-token\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795413    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-envoy\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795458    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace4e306-cefd-462b-9276-812fa8fb1c7b-istiod-ca-cert\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795497    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-data\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795523    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqmk\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-kube-api-access-sjqmk\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.795724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.795574    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-socket\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:16.796232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.796182    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"]
Oct 29 23:14:16.895795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.870343    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-59eefa-openshift-default-dockercfg-5mnhp\""
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897304    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-envoy\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897417    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace4e306-cefd-462b-9276-812fa8fb1c7b-istiod-ca-cert\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897497    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-data\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897545    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqmk\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-kube-api-access-sjqmk\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897618    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-socket\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897660    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-podinfo\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897829    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-credential-socket\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.897905    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-certs\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.898017    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-token\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.938125    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-envoy\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.940970    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-socket\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.941616    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace4e306-cefd-462b-9276-812fa8fb1c7b-istiod-ca-cert\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.941887    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-data\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.942402    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-credential-socket\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.942660    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-certs\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.005836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:16.985401    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqmk\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-kube-api-access-sjqmk\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.007609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.007512    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-podinfo\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.057786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.036299    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-token\") pod \"e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") " pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.104633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.103376    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:14:17.104633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.103453    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:17.104633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.104372    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:17.730789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.730680    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"]
Oct 29 23:14:17.737440 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:14:17.737413    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace4e306_cefd_462b_9276_812fa8fb1c7b.slice/crio-04069da5864ff44e713e32bf2055fa5af573319134e60624822f12814f4eb965 WatchSource:0}: Error finding container 04069da5864ff44e713e32bf2055fa5af573319134e60624822f12814f4eb965: Status 404 returned error can't find the container with id 04069da5864ff44e713e32bf2055fa5af573319134e60624822f12814f4eb965
Oct 29 23:14:17.740454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.740418    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:17.740564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.740491    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:17.740564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:17.740525    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:18.323371 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.323336    4827 generic.go:356] "Generic (PLEG): container finished" podID="6101a935-9dd1-4075-86ba-087b6c996633" containerID="98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f" exitCode=0
Oct 29 23:14:18.323941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.323410    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" event={"ID":"6101a935-9dd1-4075-86ba-087b6c996633","Type":"ContainerDied","Data":"98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f"}
Oct 29 23:14:18.324046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.324027    4827 scope.go:122] "RemoveContainer" containerID="98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f"
Oct 29 23:14:18.325629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.325606    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" event={"ID":"ace4e306-cefd-462b-9276-812fa8fb1c7b","Type":"ContainerStarted","Data":"85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4"}
Oct 29 23:14:18.325756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.325648    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" event={"ID":"ace4e306-cefd-462b-9276-812fa8fb1c7b","Type":"ContainerStarted","Data":"04069da5864ff44e713e32bf2055fa5af573319134e60624822f12814f4eb965"}
Oct 29 23:14:18.326287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.326255    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:18.326375 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.326345    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:18.326440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.326398    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:18.437043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:18.436985    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podStartSLOduration=2.436967139 podStartE2EDuration="2.436967139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:18.425688121 +0000 UTC m=+1894.048599144" watchObservedRunningTime="2026-10-29 23:14:18.436967139 +0000 UTC m=+1894.059878160"
Oct 29 23:14:19.105782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.105749    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:19.111950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.111928    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:19.174699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.174658    4827 patch_prober.go:29] interesting pod/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:14:19.174913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.174710    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:14:19.335953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.335921    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" event={"ID":"6101a935-9dd1-4075-86ba-087b6c996633","Type":"ContainerStarted","Data":"6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857"}
Oct 29 23:14:19.336427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.336243    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:19.337451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.337429    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:14:19.834666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.834629    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:14:19.834895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.834842    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 29 23:14:19.834954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:19.834908    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 29 23:14:20.834781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:20.834687    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 29 23:14:20.834781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:20.834753    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 29 23:14:21.230169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.230138    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:14:21.331219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331181    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-certs\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.331418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331233    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-podinfo\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.331418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331258    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-kube-api-access-xxhcd\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-kube-api-access-xxhcd\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.331418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331280    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-data\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.331418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331399    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-socket\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.331677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331434    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-credential-socket\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.331677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331506    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-certs" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:21.331677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331613    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-envoy\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.331919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.331733    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istiod-ca-cert\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.332939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.332023    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-credential-socket" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:21.332939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.332133    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-token\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-token\") pod \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\" (UID: \"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc\") "
Oct 29 23:14:21.332939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.332416    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istiod-ca-cert" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:21.332939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.332500    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-socket" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:21.332939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.332908    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-data" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:21.333311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.333243    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.333311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.333265    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.333311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.333279    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.333311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.333292    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.333525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.333308    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.334207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.334180    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-podinfo" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:14:21.336025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.336001    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-token" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:21.336025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.336012    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-envoy" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:21.336164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.336042    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-kube-api-access-xxhcd" pod "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" (UID: "c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc"). InnerVolumeSpecName "kube-api-access-xxhcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:21.357632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.357604    4827 generic.go:356] "Generic (PLEG): container finished" podID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerID="2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516" exitCode=0
Oct 29 23:14:21.357816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.357674    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"
Oct 29 23:14:21.357816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.357732    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" event={"ID":"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc","Type":"ContainerDied","Data":"2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516"}
Oct 29 23:14:21.357957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.357820    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs" event={"ID":"c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc","Type":"ContainerDied","Data":"71cec9968d8f1edfd9485f01c2170f29a2563e5c4a4657b5d78ea7eea4c46ed2"}
Oct 29 23:14:21.357957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.357855    4827 scope.go:122] "RemoveContainer" containerID="2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516"
Oct 29 23:14:21.398351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.398326    4827 scope.go:122] "RemoveContainer" containerID="a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b"
Oct 29 23:14:21.412564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.412546    4827 scope.go:122] "RemoveContainer" containerID="2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516"
Oct 29 23:14:21.412895 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:14:21.412874    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516\": container with ID starting with 2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516 not found: ID does not exist" containerID="2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516"
Oct 29 23:14:21.413036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.412940    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516"} err="failed to get container status \"2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516\": rpc error: code = NotFound desc = could not find container \"2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516\": container with ID starting with 2038eea5fade7ab95f844993c30bcbcdf50ca5eb3d6e6bc5243cf33acb949516 not found: ID does not exist"
Oct 29 23:14:21.413036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.412968    4827 scope.go:122] "RemoveContainer" containerID="a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b"
Oct 29 23:14:21.413270 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:14:21.413237    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b\": container with ID starting with a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b not found: ID does not exist" containerID="a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b"
Oct 29 23:14:21.413353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.413272    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b"} err="failed to get container status \"a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b\": rpc error: code = NotFound desc = could not find container \"a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b\": container with ID starting with a4d47106ae2a34256231678915c75435e8fac96440d51de5b69bc70e43831e4b not found: ID does not exist"
Oct 29 23:14:21.435757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.435730    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.435890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.435759    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.435890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.435787    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhcd\" (UniqueName: \"kubernetes.io/projected/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-kube-api-access-xxhcd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.435890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.435800    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:14:21.441521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.441496    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"]
Oct 29 23:14:21.447410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.447358    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-be0401-openshift-default-8f756b549-xvnjs"]
Oct 29 23:14:21.835118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.835040    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 29 23:14:21.835118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:21.835097    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 29 23:14:22.498276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:22.498246    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" path="/var/lib/kubelet/pods/c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc/volumes"
Oct 29 23:14:22.834737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:22.834666    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 29 23:14:22.834894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:22.834728    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 29 23:14:23.834971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:23.834939    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 29 23:14:23.835425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:23.834997    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 29 23:14:24.835386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:24.835349    4827 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 29 23:14:24.835952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:24.835401    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 29 23:14:25.839103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:25.839074    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:14:25.839634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:25.839612    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:14:25.840374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:25.840355    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:14:32.830003 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:14:32.829940    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:14:48.696279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.696244    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"]
Oct 29 23:14:48.697326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.697304    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy"
Oct 29 23:14:48.697326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.697326    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy"
Oct 29 23:14:48.697628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.697549    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy"
Oct 29 23:14:48.697628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.697571    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy"
Oct 29 23:14:48.701844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.701824    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.720620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.720591    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"]
Oct 29 23:14:48.725709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.725682    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-49d87f-serving-cert\""
Oct 29 23:14:48.737716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.737685    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-metrics-certs\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.737879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.737758    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-maas-api-tls\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.738061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.738038    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.738812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.738163    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkl9v\" (UniqueName: \"kubernetes.io/projected/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-kube-api-access-dkl9v\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.839297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.839268    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-metrics-certs\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.839492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.839320    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-maas-api-tls\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.839492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.839474    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.839606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.839592    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkl9v\" (UniqueName: \"kubernetes.io/projected/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-kube-api-access-dkl9v\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.840201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.840178    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.842028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.841995    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-maas-api-tls\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.842223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.842203    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-metrics-certs\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:48.862841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:48.862814    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkl9v\" (UniqueName: \"kubernetes.io/projected/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-kube-api-access-dkl9v\") pod \"maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:49.028841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.026071    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:49.444522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.444491    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"]
Oct 29 23:14:49.444738 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:14:49.444715    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa4f7dc_0517_47d3_8a31_b424b9cf7bbe.slice/crio-8353bba59264a6f955d3203e9d16d2aee63fe001b77fea699b62b0bb81d78a9b WatchSource:0}: Error finding container 8353bba59264a6f955d3203e9d16d2aee63fe001b77fea699b62b0bb81d78a9b: Status 404 returned error can't find the container with id 8353bba59264a6f955d3203e9d16d2aee63fe001b77fea699b62b0bb81d78a9b
Oct 29 23:14:49.446276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.446257    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:14:49.723537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.723453    4827 scope.go:122] "RemoveContainer" containerID="f94e5338d05d51e5943585592740d427cd946492b3a3ea2b0c6cc679334d0baa"
Oct 29 23:14:49.734446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.734421    4827 scope.go:122] "RemoveContainer" containerID="f9e9b4d089ae048a28e8ed09c8782fb2f554e4380114537dbe868f16bbb3ea60"
Oct 29 23:14:49.744593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.744573    4827 scope.go:122] "RemoveContainer" containerID="c28595134d4ba87a1a79aa368a2a7a96bf62e8754049e39c6537a589a726617c"
Oct 29 23:14:49.838044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.838005    4827 scope.go:122] "RemoveContainer" containerID="65c453f407d9eb76c2a01faf5ea947387abea7f6aeb7a42cc0ae6b1e231f6b30"
Oct 29 23:14:49.872505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.872470    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp" event={"ID":"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe","Type":"ContainerStarted","Data":"8353bba59264a6f955d3203e9d16d2aee63fe001b77fea699b62b0bb81d78a9b"}
Oct 29 23:14:49.874549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:49.874531    4827 scope.go:122] "RemoveContainer" containerID="53c4dc275adf6a3778f2cd408e363db605bcfc143c94fc60f96af313705475ce"
Oct 29 23:14:50.948792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:50.938136    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp" event={"ID":"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe","Type":"ContainerStarted","Data":"8b8c7d248d713dde69b0a6bc332055f408b7a82321d0290944f56f623e470b2f"}
Oct 29 23:14:50.948792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:50.939861    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:52.982856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:52.980903    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:52.991945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:52.991215    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:53.141721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:53.128838    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:54.052787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.051929    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp" podStartSLOduration=5.624864041 podStartE2EDuration="6.051908321s" totalImagesPullingTime="427.04428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:50.994133895 +0000 UTC m=+1926.617044913" watchObservedRunningTime="2026-10-29 23:14:54.051908321 +0000 UTC m=+1929.674819342"
Oct 29 23:14:54.120824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.116899    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"]
Oct 29 23:14:54.120824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.118320    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy"
Oct 29 23:14:54.120824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.118341    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e1ead8-0d8b-4e29-9b8a-8d2692ecb3fc" containerName="istio-proxy"
Oct 29 23:14:54.138891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.126511    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.151805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.147934    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-462fb2-openshift-default-dockercfg-cknsn\""
Oct 29 23:14:54.338637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.338549    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"]
Oct 29 23:14:54.345887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.343643    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-token\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.352820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.346178    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5745r\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-kube-api-access-5745r\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.377566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.377530    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istiod-ca-cert\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.377894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.377869    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-data\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.378079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.378062    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-podinfo\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.378245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.378228    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-envoy\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.378360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.378347    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-socket\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.378614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.378594    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-credential-socket\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.378844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.378825    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-certs\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.750755    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istiod-ca-cert\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.750888    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-data\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.750945    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-podinfo\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.751012    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-envoy\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.751046    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-socket\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.751214    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-credential-socket\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.751319    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-certs\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.751399    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-token\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.751477    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5745r\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-kube-api-access-5745r\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.773526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.753942    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istiod-ca-cert\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.793792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.776080    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-data\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.865091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.799631    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-credential-socket\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.865091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.799944    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-certs\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.925827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.918566    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-podinfo\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:54.981555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:54.981524    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-socket\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:55.039270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.039190    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-envoy\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:55.043398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.043369    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-token\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:55.045249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.045195    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"]
Oct 29 23:14:55.045496 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.045442    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"]
Oct 29 23:14:55.045632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.045619    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"]
Oct 29 23:14:55.054177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.054122    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy" containerID="cri-o://a98304eff68928584be71b579c42c7281c56901b6eab77c32665022504f4b281" gracePeriod=30
Oct 29 23:14:55.073383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.072642    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy" containerID="cri-o://9def025358a96bc7f6fd785148ae8b1c7c248db4561821c59815875eca0dde3f" gracePeriod=30
Oct 29 23:14:55.093468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.085308    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy" containerID="cri-o://29d83d68183830519d0dde6beee5be7dd9624bc04c91a2d3350fa23a740ffafc" gracePeriod=30
Oct 29 23:14:55.124966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.124449    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5745r\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-kube-api-access-5745r\") pod \"e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") " pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:55.408689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.408652    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:55.826336 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:14:55.826307    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ab563d_8ddf_4d47_a003_6aa74a8b9ad1.slice/crio-ead8281b806fbcdc48fa94222aad9da3d241d568060bfd41996e7e55390bcba3 WatchSource:0}: Error finding container ead8281b806fbcdc48fa94222aad9da3d241d568060bfd41996e7e55390bcba3: Status 404 returned error can't find the container with id ead8281b806fbcdc48fa94222aad9da3d241d568060bfd41996e7e55390bcba3
Oct 29 23:14:55.827255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.827232    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"]
Oct 29 23:14:55.829812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.829738    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:55.829910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.829851    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:55.829910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:55.829891    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:14:56.308762 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:56.307218    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" event={"ID":"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1","Type":"ContainerStarted","Data":"27e67eb0e647f75e3d148330e20a1bb6e3253dccb8ce6d11ace6e36b53f0eec5"}
Oct 29 23:14:56.308762 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:56.307319    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" event={"ID":"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1","Type":"ContainerStarted","Data":"ead8281b806fbcdc48fa94222aad9da3d241d568060bfd41996e7e55390bcba3"}
Oct 29 23:14:56.403785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:56.403685    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podStartSLOduration=2.403664035 podStartE2EDuration="2.403664035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:56.357664631 +0000 UTC m=+1931.980575652" watchObservedRunningTime="2026-10-29 23:14:56.403664035 +0000 UTC m=+1932.026575055"
Oct 29 23:14:56.409195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:56.409165    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:56.412255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:56.412219    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:14:56.412389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:56.412295    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 29 23:14:57.414489 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:57.414407    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:58.049792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:58.048113    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:14:58.324013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:58.323934    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:14:58.324989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:14:58.324968    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:15:00.206075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.204961    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"]
Oct 29 23:15:00.235786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.233965    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"]
Oct 29 23:15:00.243445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.242556    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.248220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.248141    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.249347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.249300    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"]
Oct 29 23:15:00.265706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.265678    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.266040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.266014    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:15:00.324376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.312058    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"]
Oct 29 23:15:00.327836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.327805    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"]
Oct 29 23:15:00.350601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.347985    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"]
Oct 29 23:15:00.350601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.348040    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"]
Oct 29 23:15:00.350601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.348137    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.353401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.352952    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"]
Oct 29 23:15:00.359803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.359672    4827 generic.go:356] "Generic (PLEG): container finished" podID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerID="9def025358a96bc7f6fd785148ae8b1c7c248db4561821c59815875eca0dde3f" exitCode=0
Oct 29 23:15:00.359950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.359895    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" event={"ID":"b5aca5ed-64a2-4f72-9d40-02130ffc176e","Type":"ContainerDied","Data":"9def025358a96bc7f6fd785148ae8b1c7c248db4561821c59815875eca0dde3f"}
Oct 29 23:15:00.364786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.364727    4827 generic.go:356] "Generic (PLEG): container finished" podID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerID="29d83d68183830519d0dde6beee5be7dd9624bc04c91a2d3350fa23a740ffafc" exitCode=0
Oct 29 23:15:00.364920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.364836    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" event={"ID":"3e841572-ee8f-416c-aa8f-2eeeb348423e","Type":"ContainerDied","Data":"29d83d68183830519d0dde6beee5be7dd9624bc04c91a2d3350fa23a740ffafc"}
Oct 29 23:15:00.371833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.370470    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"]
Oct 29 23:15:00.375852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.374718    4827 generic.go:356] "Generic (PLEG): container finished" podID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerID="a98304eff68928584be71b579c42c7281c56901b6eab77c32665022504f4b281" exitCode=0
Oct 29 23:15:00.389430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.388161    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:15:00.389430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.388212    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:15:00.396102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.396014    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" event={"ID":"d6713f2b-a0be-46ed-a304-513aef0121dc","Type":"ContainerDied","Data":"a98304eff68928584be71b579c42c7281c56901b6eab77c32665022504f4b281"}
Oct 29 23:15:00.396102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.396062    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"]
Oct 29 23:15:00.398576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.398430    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.404898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.404869    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"]
Oct 29 23:15:00.405029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.404916    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"]
Oct 29 23:15:00.405029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.405002    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.406990    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.407074    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d21edd-2777-4885-b045-3c012d00e931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.407164    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc1e99-60da-4a64-bceb-22bab429e457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.407279    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfpj2\" (UniqueName: \"kubernetes.io/projected/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-kube-api-access-zfpj2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.407337    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nqdf\" (UniqueName: \"kubernetes.io/projected/2fdc1e99-60da-4a64-bceb-22bab429e457-kube-api-access-6nqdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.408787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.407480    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z2mq\" (UniqueName: \"kubernetes.io/projected/38d21edd-2777-4885-b045-3c012d00e931-kube-api-access-8z2mq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.513951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.513914    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z2mq\" (UniqueName: \"kubernetes.io/projected/38d21edd-2777-4885-b045-3c012d00e931-kube-api-access-8z2mq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.514209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.514188    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.514363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.514346    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xm7\" (UniqueName: \"kubernetes.io/projected/9bccacf6-8e28-4367-88c3-48382b468dda-kube-api-access-x9xm7\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.515197 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.515172    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ztj\" (UniqueName: \"kubernetes.io/projected/4256f4cf-13ca-4286-91a8-485455a8694d-kube-api-access-87ztj\") pod \"maas-api-key-cleanup-29888595-sx7sd\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.515407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.515392    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.515540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.515523    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccacf6-8e28-4367-88c3-48382b468dda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.515717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.515701    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d21edd-2777-4885-b045-3c012d00e931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.515849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.515833    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4256f4cf-13ca-4286-91a8-485455a8694d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-sx7sd\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.515985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.515967    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67s4q\" (UniqueName: \"kubernetes.io/projected/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-kube-api-access-67s4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.516165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.516147    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc1e99-60da-4a64-bceb-22bab429e457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.516388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.516366    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfpj2\" (UniqueName: \"kubernetes.io/projected/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-kube-api-access-zfpj2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.516590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.516572    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nqdf\" (UniqueName: \"kubernetes.io/projected/2fdc1e99-60da-4a64-bceb-22bab429e457-kube-api-access-6nqdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.518566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.518263    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.519397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.519375    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc1e99-60da-4a64-bceb-22bab429e457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.521808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.521785    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d21edd-2777-4885-b045-3c012d00e931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.545013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.544563    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z2mq\" (UniqueName: \"kubernetes.io/projected/38d21edd-2777-4885-b045-3c012d00e931-kube-api-access-8z2mq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.547335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.547282    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nqdf\" (UniqueName: \"kubernetes.io/projected/2fdc1e99-60da-4a64-bceb-22bab429e457-kube-api-access-6nqdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.550601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.550553    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfpj2\" (UniqueName: \"kubernetes.io/projected/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-kube-api-access-zfpj2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.593966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.593791    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.619217    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.619282    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xm7\" (UniqueName: \"kubernetes.io/projected/9bccacf6-8e28-4367-88c3-48382b468dda-kube-api-access-x9xm7\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.619420    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ztj\" (UniqueName: \"kubernetes.io/projected/4256f4cf-13ca-4286-91a8-485455a8694d-kube-api-access-87ztj\") pod \"maas-api-key-cleanup-29888595-sx7sd\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.619510    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccacf6-8e28-4367-88c3-48382b468dda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.619608    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4256f4cf-13ca-4286-91a8-485455a8694d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-sx7sd\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.619643    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67s4q\" (UniqueName: \"kubernetes.io/projected/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-kube-api-access-67s4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.620611    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.621190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.621137    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccacf6-8e28-4367-88c3-48382b468dda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.621805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.621631    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4256f4cf-13ca-4286-91a8-485455a8694d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-sx7sd\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.650010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.648620    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ztj\" (UniqueName: \"kubernetes.io/projected/4256f4cf-13ca-4286-91a8-485455a8694d-kube-api-access-87ztj\") pod \"maas-api-key-cleanup-29888595-sx7sd\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.650010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.649640    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67s4q\" (UniqueName: \"kubernetes.io/projected/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-kube-api-access-67s4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.650428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.650391    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xm7\" (UniqueName: \"kubernetes.io/projected/9bccacf6-8e28-4367-88c3-48382b468dda-kube-api-access-x9xm7\") pod \"maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.651799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.651704    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:15:00.654605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.654585    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:15:00.721495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.721461    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-podinfo\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.721694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.721523    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-socket\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.721694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.721546    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-token\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-token\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.721694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.721595    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-certs\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.721896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.721810    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6713f2b-a0be-46ed-a304-513aef0121dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6713f2b-a0be-46ed-a304-513aef0121dc-istiod-ca-cert\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.721896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.721862    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-credential-socket\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.722024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.721938    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-envoy\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.722024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.722013    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-data\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.722129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.722043    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-kube-api-access-txjtj\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-kube-api-access-txjtj\") pod \"d6713f2b-a0be-46ed-a304-513aef0121dc\" (UID: \"d6713f2b-a0be-46ed-a304-513aef0121dc\") "
Oct 29 23:15:00.722789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.722437    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-certs" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.722789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.722724    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.723161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.723138    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6713f2b-a0be-46ed-a304-513aef0121dc-istiod-ca-cert" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:00.723416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.723394    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-credential-socket" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.724078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.724053    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:00.724934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.724888    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-socket" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.725505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.725400    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-data" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.725505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.725467    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-podinfo" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:00.729857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.729825    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-kube-api-access-txjtj" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "kube-api-access-txjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:00.733084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.733047    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-envoy" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.736130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.736103    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-token" pod "d6713f2b-a0be-46ed-a304-513aef0121dc" (UID: "d6713f2b-a0be-46ed-a304-513aef0121dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:00.751980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.751944    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:00.762335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.762297    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:00.802041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.802013    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:00.813795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.813569    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.824625    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-certs\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.824797    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-socket\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.824911    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-socket\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.824949    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-podinfo\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.824999    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-data\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825059    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-envoy\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825132    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-credential-socket\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825215    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-envoy\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825308    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-kube-api-access-6wcdj\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-kube-api-access-6wcdj\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825352    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-credential-socket\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825463    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-podinfo\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825508    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e841572-ee8f-416c-aa8f-2eeeb348423e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e841572-ee8f-416c-aa8f-2eeeb348423e-istiod-ca-cert\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825540    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-kube-api-access-kddsx\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-kube-api-access-kddsx\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825576    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-token\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-token\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825650    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-token\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-token\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825733    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-certs\") pod \"3e841572-ee8f-416c-aa8f-2eeeb348423e\" (UID: \"3e841572-ee8f-416c-aa8f-2eeeb348423e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825790    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istiod-ca-cert\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.827370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.825835    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-data\") pod \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\" (UID: \"b5aca5ed-64a2-4f72-9d40-02130ffc176e\") "
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828700    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828728    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828745    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828759    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6713f2b-a0be-46ed-a304-513aef0121dc-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828792    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828810    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828825    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6713f2b-a0be-46ed-a304-513aef0121dc-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828839    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txjtj\" (UniqueName: \"kubernetes.io/projected/d6713f2b-a0be-46ed-a304-513aef0121dc-kube-api-access-txjtj\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.828979    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.830851    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-certs" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.831330    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-credential-socket" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.831815    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e841572-ee8f-416c-aa8f-2eeeb348423e-istiod-ca-cert" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:00.832696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.832349    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istiod-ca-cert" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:00.838240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.834340    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-podinfo" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:00.838240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.834485    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-data" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.838240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.834604    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-socket" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.838240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.834716    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-certs" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.838240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.834865    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-data" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.838240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.834875    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-socket" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.838240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.835060    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-credential-socket" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.839329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.839291    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-envoy" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.840150 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.840121    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-token" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:00.857120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.856927    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-podinfo" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:00.857120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.857017    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-token" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:00.857120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.857065    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-kube-api-access-kddsx" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "kube-api-access-kddsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:00.869403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.858885    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-kube-api-access-6wcdj" pod "3e841572-ee8f-416c-aa8f-2eeeb348423e" (UID: "3e841572-ee8f-416c-aa8f-2eeeb348423e"). InnerVolumeSpecName "kube-api-access-6wcdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:00.885713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.884930    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-envoy" pod "b5aca5ed-64a2-4f72-9d40-02130ffc176e" (UID: "b5aca5ed-64a2-4f72-9d40-02130ffc176e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:00.956057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.956030    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.956310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.956294    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.956425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.956412    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.956543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.956527    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.958663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.958641    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959017    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959171    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959290    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959405    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcdj\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-kube-api-access-6wcdj\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959527    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959640    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959814    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e841572-ee8f-416c-aa8f-2eeeb348423e-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.959965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.959953    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kddsx\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-kube-api-access-kddsx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.960072 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.960060    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.960171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.960161    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e841572-ee8f-416c-aa8f-2eeeb348423e-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.960275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.960265    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e841572-ee8f-416c-aa8f-2eeeb348423e-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.960384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.960373    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:00.960487 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:00.960477    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aca5ed-64a2-4f72-9d40-02130ffc176e-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:01.290858 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:01.290825    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdc1e99_60da_4a64_bceb_22bab429e457.slice/crio-28c62f2f3a4f13fb5cb5e26b6c60020fd32984edb0ae1a8acb5c14b799f0a9ca WatchSource:0}: Error finding container 28c62f2f3a4f13fb5cb5e26b6c60020fd32984edb0ae1a8acb5c14b799f0a9ca: Status 404 returned error can't find the container with id 28c62f2f3a4f13fb5cb5e26b6c60020fd32984edb0ae1a8acb5c14b799f0a9ca
Oct 29 23:15:01.298963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.298918    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"]
Oct 29 23:15:01.316044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.312636    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy"
Oct 29 23:15:01.316044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.312667    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy"
Oct 29 23:15:01.319103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.316336    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy"
Oct 29 23:15:01.319103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.316360    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy"
Oct 29 23:15:01.319103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.316391    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy"
Oct 29 23:15:01.319103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.316400    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy"
Oct 29 23:15:01.319103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.316641    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" containerName="istio-proxy"
Oct 29 23:15:01.319103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.316660    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" containerName="istio-proxy"
Oct 29 23:15:01.319103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.316694    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" containerName="istio-proxy"
Oct 29 23:15:01.331646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.331429    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"]
Oct 29 23:15:01.331646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.331485    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"]
Oct 29 23:15:01.332567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.331881    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:01.336711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.336692    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"]
Oct 29 23:15:01.355507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.355479    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"]
Oct 29 23:15:01.422255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.402547    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-462fb2\""
Oct 29 23:15:01.540856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.538705    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063e478-02e9-4920-a461-50223a414968-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:01.540856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.538830    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl6jc\" (UniqueName: \"kubernetes.io/projected/2063e478-02e9-4920-a461-50223a414968-kube-api-access-sl6jc\") pod \"payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:01.544811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.543256    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-462fb2-dockercfg-mxjvw\""
Oct 29 23:15:01.833834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.833680    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"]
Oct 29 23:15:01.853354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.851981    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"]
Oct 29 23:15:01.940830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.930643    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"]
Oct 29 23:15:01.940830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.931363    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063e478-02e9-4920-a461-50223a414968-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:01.940830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.931498    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl6jc\" (UniqueName: \"kubernetes.io/projected/2063e478-02e9-4920-a461-50223a414968-kube-api-access-sl6jc\") pod \"payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:01.940830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.933174    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063e478-02e9-4920-a461-50223a414968-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:01.992257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:01.991430    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl6jc\" (UniqueName: \"kubernetes.io/projected/2063e478-02e9-4920-a461-50223a414968-kube-api-access-sl6jc\") pod \"payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:02.011074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.011038    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" event={"ID":"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91","Type":"ContainerStarted","Data":"42a378fe79b4112fbf75594e4870db78145c7834e30e28762ec992a459a3ef27"}
Oct 29 23:15:02.018636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.017023    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d" event={"ID":"3e841572-ee8f-416c-aa8f-2eeeb348423e","Type":"ContainerDied","Data":"7d202e04dff174a8300509fb9980bad55387d2418ad2eed269180a6dd8b281d4"}
Oct 29 23:15:02.018636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.017087    4827 scope.go:122] "RemoveContainer" containerID="29d83d68183830519d0dde6beee5be7dd9624bc04c91a2d3350fa23a740ffafc"
Oct 29 23:15:02.018636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.017260    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"
Oct 29 23:15:02.023469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.023450    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"
Oct 29 23:15:02.023749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.023472    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4" event={"ID":"d6713f2b-a0be-46ed-a304-513aef0121dc","Type":"ContainerDied","Data":"9c4a377e5fdc1298678bbad4c31ecbc0b0ba10c6cd5f3fca69e63ec8d8226bc7"}
Oct 29 23:15:02.025504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.025467    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" event={"ID":"38d21edd-2777-4885-b045-3c012d00e931","Type":"ContainerStarted","Data":"eba27cd7b058d9e13354e75d8b488287dd7972bde20140a58aedd65ac794b983"}
Oct 29 23:15:02.029725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.029684    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"
Oct 29 23:15:02.033855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.033833    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n" event={"ID":"b5aca5ed-64a2-4f72-9d40-02130ffc176e","Type":"ContainerDied","Data":"67022a2ba9432f194df222f94195142ce7e9606891cc5d0d23c8b7d05e5d2614"}
Oct 29 23:15:02.036174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.036146    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" event={"ID":"2fdc1e99-60da-4a64-bceb-22bab429e457","Type":"ContainerStarted","Data":"28c62f2f3a4f13fb5cb5e26b6c60020fd32984edb0ae1a8acb5c14b799f0a9ca"}
Oct 29 23:15:02.054082 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.053956    4827 scope.go:122] "RemoveContainer" containerID="a98304eff68928584be71b579c42c7281c56901b6eab77c32665022504f4b281"
Oct 29 23:15:02.081518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.081492    4827 scope.go:122] "RemoveContainer" containerID="9def025358a96bc7f6fd785148ae8b1c7c248db4561821c59815875eca0dde3f"
Oct 29 23:15:02.107720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.107691    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:02.114716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.114658    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" podStartSLOduration=2.114639493 podStartE2EDuration="2.114639493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:02.109793332 +0000 UTC m=+1937.732704348" watchObservedRunningTime="2026-10-29 23:15:02.114639493 +0000 UTC m=+1937.737550514"
Oct 29 23:15:02.185203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.185166    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"]
Oct 29 23:15:02.190193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.190162    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-jvhs4"]
Oct 29 23:15:02.205002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.204969    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"]
Oct 29 23:15:02.211843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.210970    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-wzx9n"]
Oct 29 23:15:02.247547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.245941    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"]
Oct 29 23:15:02.259225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.259172    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-7qx2d"]
Oct 29 23:15:02.493881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.493854    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e841572-ee8f-416c-aa8f-2eeeb348423e" path="/var/lib/kubelet/pods/3e841572-ee8f-416c-aa8f-2eeeb348423e/volumes"
Oct 29 23:15:02.494651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.494628    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5aca5ed-64a2-4f72-9d40-02130ffc176e" path="/var/lib/kubelet/pods/b5aca5ed-64a2-4f72-9d40-02130ffc176e/volumes"
Oct 29 23:15:02.495250 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.495230    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6713f2b-a0be-46ed-a304-513aef0121dc" path="/var/lib/kubelet/pods/d6713f2b-a0be-46ed-a304-513aef0121dc/volumes"
Oct 29 23:15:02.496047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:02.496028    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"]
Oct 29 23:15:02.570320 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:02.570284    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2063e478_02e9_4920_a461_50223a414968.slice/crio-aaf246d62830cbeb5f311d116dd3e0f15ee64f99f788f14b09c78a746416e567 WatchSource:0}: Error finding container aaf246d62830cbeb5f311d116dd3e0f15ee64f99f788f14b09c78a746416e567: Status 404 returned error can't find the container with id aaf246d62830cbeb5f311d116dd3e0f15ee64f99f788f14b09c78a746416e567
Oct 29 23:15:02.839518 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:02.839488    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:15:03.047823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.047710    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerStarted","Data":"9333f54e5ce77d2be25ab0cb4d9b9518d6725f3798b4e98d5ab34b00a2c05f4a"}
Oct 29 23:15:03.047823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.047809    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerStarted","Data":"0deea2e9f26784358c939c1d0b5e7036ec974d32a2429506d0199a9ca63eb0c3"}
Oct 29 23:15:03.049304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.049278    4827 generic.go:356] "Generic (PLEG): container finished" podID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerID="f8a7d910e66bb5e43fa4cb150951d7c6532a1926cf6911cafa5959a3c54a240b" exitCode=60
Oct 29 23:15:03.049452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.049348    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" event={"ID":"2fdc1e99-60da-4a64-bceb-22bab429e457","Type":"ContainerDied","Data":"f8a7d910e66bb5e43fa4cb150951d7c6532a1926cf6911cafa5959a3c54a240b"}
Oct 29 23:15:03.049659 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.049641    4827 scope.go:122] "RemoveContainer" containerID="f8a7d910e66bb5e43fa4cb150951d7c6532a1926cf6911cafa5959a3c54a240b"
Oct 29 23:15:03.051146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.051123    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" event={"ID":"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91","Type":"ContainerStarted","Data":"631eb8a61c36ae3f005303fc54fe6e8db91f7d694db5c758f8a353c86853d2df"}
Oct 29 23:15:03.055356 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.055320    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" event={"ID":"2063e478-02e9-4920-a461-50223a414968","Type":"ContainerStarted","Data":"43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b"}
Oct 29 23:15:03.055555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.055529    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" event={"ID":"2063e478-02e9-4920-a461-50223a414968","Type":"ContainerStarted","Data":"aaf246d62830cbeb5f311d116dd3e0f15ee64f99f788f14b09c78a746416e567"}
Oct 29 23:15:03.055673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.055580    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:03.057192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.057173    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" event={"ID":"38d21edd-2777-4885-b045-3c012d00e931","Type":"ContainerStarted","Data":"5f5d7a074d8630b8e81eec739637f7dcc46c5e78531e3a0731d049b23ebaeccf"}
Oct 29 23:15:03.058914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.058670    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerStarted","Data":"8b5bb6d0c2dfef1e8c68f97d0a282998e381e1be7b289b10319f348f881b0c80"}
Oct 29 23:15:03.058914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.058714    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerStarted","Data":"cfa5332cec5d66782e39b9b583ecd3f86866837a556bbf011340a538c119a0e5"}
Oct 29 23:15:03.060272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.060253    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" event={"ID":"9bccacf6-8e28-4367-88c3-48382b468dda","Type":"ContainerStarted","Data":"9febd2afd878465f1f10bd87e1703477696710f34ff0acc02418349f870ed173"}
Oct 29 23:15:03.060360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.060286    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" event={"ID":"9bccacf6-8e28-4367-88c3-48382b468dda","Type":"ContainerStarted","Data":"12f382abb54b22918805edb01c5a9cf1c6e9c9c7e44d60485c9d0c5d8e05ee26"}
Oct 29 23:15:03.094424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.094375    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" podStartSLOduration=3.09436174 podStartE2EDuration="3.09436174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.089679613 +0000 UTC m=+1938.712590633" watchObservedRunningTime="2026-10-29 23:15:03.09436174 +0000 UTC m=+1938.717272759"
Oct 29 23:15:03.170318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.170239    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" podStartSLOduration=3.170218545 podStartE2EDuration="3.170218545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.147103179 +0000 UTC m=+1938.770014199" watchObservedRunningTime="2026-10-29 23:15:03.170218545 +0000 UTC m=+1938.793129565"
Oct 29 23:15:03.193737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.193673    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" podStartSLOduration=2.193653923 podStartE2EDuration="2.193653923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.187071766 +0000 UTC m=+1938.809982787" watchObservedRunningTime="2026-10-29 23:15:03.193653923 +0000 UTC m=+1938.816564945"
Oct 29 23:15:03.231986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.231934    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" podStartSLOduration=3.231921028 podStartE2EDuration="3.231921028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.226577978 +0000 UTC m=+1938.849488997" watchObservedRunningTime="2026-10-29 23:15:03.231921028 +0000 UTC m=+1938.854832048"
Oct 29 23:15:03.309184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.309098    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" podStartSLOduration=3.309080484 podStartE2EDuration="3.309080484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.308661716 +0000 UTC m=+1938.931572736" watchObservedRunningTime="2026-10-29 23:15:03.309080484 +0000 UTC m=+1938.931991504"
Oct 29 23:15:03.354526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.354477    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" podStartSLOduration=3.354464325 podStartE2EDuration="3.354464325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.350722871 +0000 UTC m=+1938.973633895" watchObservedRunningTime="2026-10-29 23:15:03.354464325 +0000 UTC m=+1938.977375348"
Oct 29 23:15:03.695730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.695698    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"]
Oct 29 23:15:03.696142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:03.695936    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy" containerID="cri-o://fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8" gracePeriod=30
Oct 29 23:15:04.134939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.134898    4827 generic.go:356] "Generic (PLEG): container finished" podID="4256f4cf-13ca-4286-91a8-485455a8694d" containerID="8b5bb6d0c2dfef1e8c68f97d0a282998e381e1be7b289b10319f348f881b0c80" exitCode=60
Oct 29 23:15:04.140579 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.135685    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerDied","Data":"8b5bb6d0c2dfef1e8c68f97d0a282998e381e1be7b289b10319f348f881b0c80"}
Oct 29 23:15:04.140784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.136795    4827 scope.go:122] "RemoveContainer" containerID="8b5bb6d0c2dfef1e8c68f97d0a282998e381e1be7b289b10319f348f881b0c80"
Oct 29 23:15:04.143847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.143817    4827 generic.go:356] "Generic (PLEG): container finished" podID="9bccacf6-8e28-4367-88c3-48382b468dda" containerID="9febd2afd878465f1f10bd87e1703477696710f34ff0acc02418349f870ed173" exitCode=60
Oct 29 23:15:04.143999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.143961    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" event={"ID":"9bccacf6-8e28-4367-88c3-48382b468dda","Type":"ContainerDied","Data":"9febd2afd878465f1f10bd87e1703477696710f34ff0acc02418349f870ed173"}
Oct 29 23:15:04.144483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.144461    4827 scope.go:122] "RemoveContainer" containerID="9febd2afd878465f1f10bd87e1703477696710f34ff0acc02418349f870ed173"
Oct 29 23:15:04.194250 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.193304    4827 generic.go:356] "Generic (PLEG): container finished" podID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerID="9333f54e5ce77d2be25ab0cb4d9b9518d6725f3798b4e98d5ab34b00a2c05f4a" exitCode=60
Oct 29 23:15:04.194250 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.193440    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerDied","Data":"9333f54e5ce77d2be25ab0cb4d9b9518d6725f3798b4e98d5ab34b00a2c05f4a"}
Oct 29 23:15:04.194250 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.193985    4827 scope.go:122] "RemoveContainer" containerID="9333f54e5ce77d2be25ab0cb4d9b9518d6725f3798b4e98d5ab34b00a2c05f4a"
Oct 29 23:15:04.257404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.256246    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"]
Oct 29 23:15:04.266413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.266395    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"]
Oct 29 23:15:04.266583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.266565    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" event={"ID":"2fdc1e99-60da-4a64-bceb-22bab429e457","Type":"ContainerStarted","Data":"175e2ad5cf6f260bbd081cac416b338575ad276bfc10722c5f3e229504c8a71a"}
Oct 29 23:15:04.266813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.266458    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.266962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.266940    4827 generic.go:356] "Generic (PLEG): container finished" podID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerID="631eb8a61c36ae3f005303fc54fe6e8db91f7d694db5c758f8a353c86853d2df" exitCode=60
Oct 29 23:15:04.267102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.267089    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" event={"ID":"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91","Type":"ContainerDied","Data":"631eb8a61c36ae3f005303fc54fe6e8db91f7d694db5c758f8a353c86853d2df"}
Oct 29 23:15:04.267376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.267359    4827 scope.go:122] "RemoveContainer" containerID="631eb8a61c36ae3f005303fc54fe6e8db91f7d694db5c758f8a353c86853d2df"
Oct 29 23:15:04.278350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.278316    4827 generic.go:356] "Generic (PLEG): container finished" podID="38d21edd-2777-4885-b045-3c012d00e931" containerID="5f5d7a074d8630b8e81eec739637f7dcc46c5e78531e3a0731d049b23ebaeccf" exitCode=60
Oct 29 23:15:04.278690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.278667    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" event={"ID":"38d21edd-2777-4885-b045-3c012d00e931","Type":"ContainerDied","Data":"5f5d7a074d8630b8e81eec739637f7dcc46c5e78531e3a0731d049b23ebaeccf"}
Oct 29 23:15:04.279414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.278886    4827 scope.go:122] "RemoveContainer" containerID="5f5d7a074d8630b8e81eec739637f7dcc46c5e78531e3a0731d049b23ebaeccf"
Oct 29 23:15:04.285644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.283419    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:15:04.371805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.371758    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.372136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.372110    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv42w\" (UniqueName: \"kubernetes.io/projected/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-kube-api-access-xv42w\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.372441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.372418    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.479278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.474828    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.479278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.474996    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv42w\" (UniqueName: \"kubernetes.io/projected/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-kube-api-access-xv42w\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.479278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.475195    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.479278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.476118    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.489822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.489738    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.551388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.551365    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv42w\" (UniqueName: \"kubernetes.io/projected/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-kube-api-access-xv42w\") pod \"maas-api-revoke-sub-8029f49f-ggmp7\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.575506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.575482    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:04.890974 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:04.890945    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e70f9e_dbc0_4f79_a139_82e07b76d9a6.slice/crio-7114c4875b5eff98f4ac67a7f0a902a13f7570ec5679809a1a43b31ff0bd4fe8 WatchSource:0}: Error finding container 7114c4875b5eff98f4ac67a7f0a902a13f7570ec5679809a1a43b31ff0bd4fe8: Status 404 returned error can't find the container with id 7114c4875b5eff98f4ac67a7f0a902a13f7570ec5679809a1a43b31ff0bd4fe8
Oct 29 23:15:04.894308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:04.894279    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"]
Oct 29 23:15:05.292220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.292190    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerStarted","Data":"832539c383e13ec094c16d9495ef53bf27851bc2b1e13a28e431060338228574"}
Oct 29 23:15:05.295132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.295104    4827 generic.go:356] "Generic (PLEG): container finished" podID="9bccacf6-8e28-4367-88c3-48382b468dda" containerID="098d61ce375acb4d062de4d901e29a3b6419ff989384ec0c26c614f0321d52f6" exitCode=60
Oct 29 23:15:05.295244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.295222    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" event={"ID":"9bccacf6-8e28-4367-88c3-48382b468dda","Type":"ContainerDied","Data":"098d61ce375acb4d062de4d901e29a3b6419ff989384ec0c26c614f0321d52f6"}
Oct 29 23:15:05.295310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.295289    4827 scope.go:122] "RemoveContainer" containerID="9febd2afd878465f1f10bd87e1703477696710f34ff0acc02418349f870ed173"
Oct 29 23:15:05.295578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.295540    4827 scope.go:122] "RemoveContainer" containerID="098d61ce375acb4d062de4d901e29a3b6419ff989384ec0c26c614f0321d52f6"
Oct 29 23:15:05.296064 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:05.296028    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s_odh-ai-gateway-infra(9bccacf6-8e28-4367-88c3-48382b468dda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" podUID="9bccacf6-8e28-4367-88c3-48382b468dda"
Oct 29 23:15:05.298639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.298612    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerStarted","Data":"c574b9198258a13335fade6b1e4aeae2636d9eab00d8ca27697a1dd47d9ff90e"}
Oct 29 23:15:05.300973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.300949    4827 generic.go:356] "Generic (PLEG): container finished" podID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerID="175e2ad5cf6f260bbd081cac416b338575ad276bfc10722c5f3e229504c8a71a" exitCode=60
Oct 29 23:15:05.301124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.301093    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" event={"ID":"2fdc1e99-60da-4a64-bceb-22bab429e457","Type":"ContainerDied","Data":"175e2ad5cf6f260bbd081cac416b338575ad276bfc10722c5f3e229504c8a71a"}
Oct 29 23:15:05.301269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.301253    4827 scope.go:122] "RemoveContainer" containerID="175e2ad5cf6f260bbd081cac416b338575ad276bfc10722c5f3e229504c8a71a"
Oct 29 23:15:05.301575 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:05.301550    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z_odh-ai-gateway-infra(2fdc1e99-60da-4a64-bceb-22bab429e457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457"
Oct 29 23:15:05.303537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.303514    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" event={"ID":"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91","Type":"ContainerStarted","Data":"aaa5a8f98180117593ae165637b05077b39418b6d305a8a4f820a436c72ba062"}
Oct 29 23:15:05.306037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.306015    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" event={"ID":"38d21edd-2777-4885-b045-3c012d00e931","Type":"ContainerStarted","Data":"a5ff390f6abc4f0d6b694d5c7ad8c624b9debcdfcc656981f725c7cb95e3c228"}
Oct 29 23:15:05.307554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.307536    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7" event={"ID":"65e70f9e-dbc0-4f79-a139-82e07b76d9a6","Type":"ContainerStarted","Data":"b1104cabeb60579e8a4fce01d2a7518e83f9e6399185559daa9409b77d5733ec"}
Oct 29 23:15:05.307646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.307576    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7" event={"ID":"65e70f9e-dbc0-4f79-a139-82e07b76d9a6","Type":"ContainerStarted","Data":"7114c4875b5eff98f4ac67a7f0a902a13f7570ec5679809a1a43b31ff0bd4fe8"}
Oct 29 23:15:05.374234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.374183    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7" podStartSLOduration=1.3741703969999999 podStartE2EDuration="1.374170397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:05.374040794 +0000 UTC m=+1940.996951814" watchObservedRunningTime="2026-10-29 23:15:05.374170397 +0000 UTC m=+1940.997081417"
Oct 29 23:15:05.402466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:05.402446    4827 scope.go:122] "RemoveContainer" containerID="f8a7d910e66bb5e43fa4cb150951d7c6532a1926cf6911cafa5959a3c54a240b"
Oct 29 23:15:06.317823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.317779    4827 generic.go:356] "Generic (PLEG): container finished" podID="65e70f9e-dbc0-4f79-a139-82e07b76d9a6" containerID="b1104cabeb60579e8a4fce01d2a7518e83f9e6399185559daa9409b77d5733ec" exitCode=0
Oct 29 23:15:06.317823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.317820    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7" event={"ID":"65e70f9e-dbc0-4f79-a139-82e07b76d9a6","Type":"ContainerDied","Data":"b1104cabeb60579e8a4fce01d2a7518e83f9e6399185559daa9409b77d5733ec"}
Oct 29 23:15:06.319592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.319560    4827 generic.go:356] "Generic (PLEG): container finished" podID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerID="50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417" exitCode=0
Oct 29 23:15:06.319748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.319645    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" event={"ID":"c38d50dd-ecfc-48ec-9dee-ea03ac789908","Type":"ContainerDied","Data":"50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417"}
Oct 29 23:15:06.320220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.320196    4827 scope.go:122] "RemoveContainer" containerID="50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417"
Oct 29 23:15:06.322018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.321997    4827 generic.go:356] "Generic (PLEG): container finished" podID="4256f4cf-13ca-4286-91a8-485455a8694d" containerID="832539c383e13ec094c16d9495ef53bf27851bc2b1e13a28e431060338228574" exitCode=60
Oct 29 23:15:06.322098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.322077    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerDied","Data":"832539c383e13ec094c16d9495ef53bf27851bc2b1e13a28e431060338228574"}
Oct 29 23:15:06.322175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.322123    4827 scope.go:122] "RemoveContainer" containerID="8b5bb6d0c2dfef1e8c68f97d0a282998e381e1be7b289b10319f348f881b0c80"
Oct 29 23:15:06.322396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.322372    4827 scope.go:122] "RemoveContainer" containerID="832539c383e13ec094c16d9495ef53bf27851bc2b1e13a28e431060338228574"
Oct 29 23:15:06.322595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.322558    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:06.322683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.322651    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:06.322751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.322698    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:06.322858 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:06.322830    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-sx7sd_odh-ai-gateway-infra(4256f4cf-13ca-4286-91a8-485455a8694d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" podUID="4256f4cf-13ca-4286-91a8-485455a8694d"
Oct 29 23:15:06.324182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.324161    4827 scope.go:122] "RemoveContainer" containerID="098d61ce375acb4d062de4d901e29a3b6419ff989384ec0c26c614f0321d52f6"
Oct 29 23:15:06.324623 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:06.324523    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s_odh-ai-gateway-infra(9bccacf6-8e28-4367-88c3-48382b468dda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" podUID="9bccacf6-8e28-4367-88c3-48382b468dda"
Oct 29 23:15:06.326209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.326184    4827 generic.go:356] "Generic (PLEG): container finished" podID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerID="c574b9198258a13335fade6b1e4aeae2636d9eab00d8ca27697a1dd47d9ff90e" exitCode=60
Oct 29 23:15:06.326320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.326255    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerDied","Data":"c574b9198258a13335fade6b1e4aeae2636d9eab00d8ca27697a1dd47d9ff90e"}
Oct 29 23:15:06.326490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.326472    4827 scope.go:122] "RemoveContainer" containerID="c574b9198258a13335fade6b1e4aeae2636d9eab00d8ca27697a1dd47d9ff90e"
Oct 29 23:15:06.326853 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:06.326831    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq_odh-ai-gateway-infra(a78ac709-b2ed-447e-94e8-c5c60e2df3a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4"
Oct 29 23:15:06.328302 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.328288    4827 scope.go:122] "RemoveContainer" containerID="175e2ad5cf6f260bbd081cac416b338575ad276bfc10722c5f3e229504c8a71a"
Oct 29 23:15:06.328644 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:06.328623    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z_odh-ai-gateway-infra(2fdc1e99-60da-4a64-bceb-22bab429e457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457"
Oct 29 23:15:06.330108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.330089    4827 generic.go:356] "Generic (PLEG): container finished" podID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerID="aaa5a8f98180117593ae165637b05077b39418b6d305a8a4f820a436c72ba062" exitCode=60
Oct 29 23:15:06.330217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.330187    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" event={"ID":"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91","Type":"ContainerDied","Data":"aaa5a8f98180117593ae165637b05077b39418b6d305a8a4f820a436c72ba062"}
Oct 29 23:15:06.330389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.330372    4827 scope.go:122] "RemoveContainer" containerID="aaa5a8f98180117593ae165637b05077b39418b6d305a8a4f820a436c72ba062"
Oct 29 23:15:06.330643 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:06.330624    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn_odh-ai-gateway-infra(4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91"
Oct 29 23:15:06.332068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.332036    4827 generic.go:356] "Generic (PLEG): container finished" podID="38d21edd-2777-4885-b045-3c012d00e931" containerID="a5ff390f6abc4f0d6b694d5c7ad8c624b9debcdfcc656981f725c7cb95e3c228" exitCode=60
Oct 29 23:15:06.332165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.332079    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" event={"ID":"38d21edd-2777-4885-b045-3c012d00e931","Type":"ContainerDied","Data":"a5ff390f6abc4f0d6b694d5c7ad8c624b9debcdfcc656981f725c7cb95e3c228"}
Oct 29 23:15:06.332368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.332351    4827 scope.go:122] "RemoveContainer" containerID="a5ff390f6abc4f0d6b694d5c7ad8c624b9debcdfcc656981f725c7cb95e3c228"
Oct 29 23:15:06.332602 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:06.332585    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2_odh-ai-gateway-infra(38d21edd-2777-4885-b045-3c012d00e931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" podUID="38d21edd-2777-4885-b045-3c012d00e931"
Oct 29 23:15:06.340742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.340694    4827 scope.go:122] "RemoveContainer" containerID="9333f54e5ce77d2be25ab0cb4d9b9518d6725f3798b4e98d5ab34b00a2c05f4a"
Oct 29 23:15:06.357293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.357273    4827 scope.go:122] "RemoveContainer" containerID="631eb8a61c36ae3f005303fc54fe6e8db91f7d694db5c758f8a353c86853d2df"
Oct 29 23:15:06.374251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:06.374226    4827 scope.go:122] "RemoveContainer" containerID="5f5d7a074d8630b8e81eec739637f7dcc46c5e78531e3a0731d049b23ebaeccf"
Oct 29 23:15:07.342619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.342583    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" event={"ID":"c38d50dd-ecfc-48ec-9dee-ea03ac789908","Type":"ContainerStarted","Data":"ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4"}
Oct 29 23:15:07.344936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.344909    4827 scope.go:122] "RemoveContainer" containerID="832539c383e13ec094c16d9495ef53bf27851bc2b1e13a28e431060338228574"
Oct 29 23:15:07.345353 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:07.345329    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-sx7sd_odh-ai-gateway-infra(4256f4cf-13ca-4286-91a8-485455a8694d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" podUID="4256f4cf-13ca-4286-91a8-485455a8694d"
Oct 29 23:15:07.346596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.346577    4827 scope.go:122] "RemoveContainer" containerID="c574b9198258a13335fade6b1e4aeae2636d9eab00d8ca27697a1dd47d9ff90e"
Oct 29 23:15:07.346942 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:07.346921    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq_odh-ai-gateway-infra(a78ac709-b2ed-447e-94e8-c5c60e2df3a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4"
Oct 29 23:15:07.348201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.348184    4827 scope.go:122] "RemoveContainer" containerID="aaa5a8f98180117593ae165637b05077b39418b6d305a8a4f820a436c72ba062"
Oct 29 23:15:07.348516 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:07.348493    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn_odh-ai-gateway-infra(4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91"
Oct 29 23:15:07.350289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.349941    4827 scope.go:122] "RemoveContainer" containerID="a5ff390f6abc4f0d6b694d5c7ad8c624b9debcdfcc656981f725c7cb95e3c228"
Oct 29 23:15:07.350289 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:07.350180    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2_odh-ai-gateway-infra(38d21edd-2777-4885-b045-3c012d00e931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" podUID="38d21edd-2777-4885-b045-3c012d00e931"
Oct 29 23:15:07.540149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.540113    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:15:07.540815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.540782    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 29 23:15:07.540960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.540842    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 29 23:15:07.619181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.619154    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:07.759713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.759664    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-openshift-ca\") pod \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") "
Oct 29 23:15:07.759932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.759730    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-kube-api-access-xv42w\" (UniqueName: \"kubernetes.io/projected/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-kube-api-access-xv42w\") pod \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") "
Oct 29 23:15:07.759932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.759903    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-xks-ca\") pod \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\" (UID: \"65e70f9e-dbc0-4f79-a139-82e07b76d9a6\") "
Oct 29 23:15:07.761459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.761431    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-openshift-ca" pod "65e70f9e-dbc0-4f79-a139-82e07b76d9a6" (UID: "65e70f9e-dbc0-4f79-a139-82e07b76d9a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:07.795963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.795928    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-kube-api-access-xv42w" pod "65e70f9e-dbc0-4f79-a139-82e07b76d9a6" (UID: "65e70f9e-dbc0-4f79-a139-82e07b76d9a6"). InnerVolumeSpecName "kube-api-access-xv42w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:07.796636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.796584    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-xks-ca" pod "65e70f9e-dbc0-4f79-a139-82e07b76d9a6" (UID: "65e70f9e-dbc0-4f79-a139-82e07b76d9a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:15:07.863296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.863206    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:07.863296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.863245    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:07.863296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:07.863263    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv42w\" (UniqueName: \"kubernetes.io/projected/65e70f9e-dbc0-4f79-a139-82e07b76d9a6-kube-api-access-xv42w\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:08.359468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:08.359440    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"
Oct 29 23:15:08.359900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:08.359478    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7" event={"ID":"65e70f9e-dbc0-4f79-a139-82e07b76d9a6","Type":"ContainerDied","Data":"7114c4875b5eff98f4ac67a7f0a902a13f7570ec5679809a1a43b31ff0bd4fe8"}
Oct 29 23:15:08.359900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:08.359523    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7114c4875b5eff98f4ac67a7f0a902a13f7570ec5679809a1a43b31ff0bd4fe8"
Oct 29 23:15:08.540960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:08.540928    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 29 23:15:08.541134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:08.540978    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 29 23:15:09.049616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.049589    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:15:09.088831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.086396    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"]
Oct 29 23:15:09.088831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.087685    4827 cpu_manager.go:414] "containerMap: removing container" podUID="65e70f9e-dbc0-4f79-a139-82e07b76d9a6" containerName="revoke-keys"
Oct 29 23:15:09.088831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.087707    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e70f9e-dbc0-4f79-a139-82e07b76d9a6" containerName="revoke-keys"
Oct 29 23:15:09.088831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.087751    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy"
Oct 29 23:15:09.088831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.087779    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy"
Oct 29 23:15:09.088831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.088041    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e70f9e-dbc0-4f79-a139-82e07b76d9a6" containerName="revoke-keys"
Oct 29 23:15:09.088831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.088063    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerName="istio-proxy"
Oct 29 23:15:09.098882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.095172    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.098882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.096844    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"]
Oct 29 23:15:09.118782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.118751    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:15:09.203396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203367    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-data\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203418    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3bda7f-64d7-41bb-9649-bba9983a995a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3bda7f-64d7-41bb-9649-bba9983a995a-istiod-ca-cert\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203484    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-envoy\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203583    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-kube-api-access-m5stn\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-kube-api-access-m5stn\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203638    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-credential-socket\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203718    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-socket\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203742    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-token\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-token\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203736    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-data" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:09.203842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203797    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-certs\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.203842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.203818    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-podinfo\") pod \"7e3bda7f-64d7-41bb-9649-bba9983a995a\" (UID: \"7e3bda7f-64d7-41bb-9649-bba9983a995a\") "
Oct 29 23:15:09.204253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204040    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8m2\" (UniqueName: \"kubernetes.io/projected/823210ce-20d4-48dc-99a3-1da8d708c6e7-kube-api-access-mt8m2\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.204253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204073    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.204253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204079    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-socket" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:09.204416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204266    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-certs" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:09.204416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204292    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3bda7f-64d7-41bb-9649-bba9983a995a-istiod-ca-cert" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:09.204522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204475    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.204966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204888    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.204966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204915    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.204966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204927    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.204966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204937    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3bda7f-64d7-41bb-9649-bba9983a995a-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.204966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.204889    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-credential-socket" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:09.206654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.206626    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-envoy" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:09.207244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.207223    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-podinfo" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:09.207344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.207266    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-token" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:09.207563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.207541    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-kube-api-access-m5stn" pod "7e3bda7f-64d7-41bb-9649-bba9983a995a" (UID: "7e3bda7f-64d7-41bb-9649-bba9983a995a"). InnerVolumeSpecName "kube-api-access-m5stn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:09.305972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.305876    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8m2\" (UniqueName: \"kubernetes.io/projected/823210ce-20d4-48dc-99a3-1da8d708c6e7-kube-api-access-mt8m2\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.305972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.305938    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.306280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.306215    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.306476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.306459    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.306578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.306480    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.306578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.306489    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.306578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.306498    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3bda7f-64d7-41bb-9649-bba9983a995a-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.306578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.306507    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5stn\" (UniqueName: \"kubernetes.io/projected/7e3bda7f-64d7-41bb-9649-bba9983a995a-kube-api-access-m5stn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.306920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.306896    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.308575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.308557    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.330049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.330028    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8m2\" (UniqueName: \"kubernetes.io/projected/823210ce-20d4-48dc-99a3-1da8d708c6e7-kube-api-access-mt8m2\") pod \"maas-api-revoke-sub-070c6d14-hgzxb\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.394556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.394518    4827 generic.go:356] "Generic (PLEG): container finished" podID="7e3bda7f-64d7-41bb-9649-bba9983a995a" containerID="fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8" exitCode=0
Oct 29 23:15:09.394556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.394567    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"
Oct 29 23:15:09.395178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.394612    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" event={"ID":"7e3bda7f-64d7-41bb-9649-bba9983a995a","Type":"ContainerDied","Data":"fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8"}
Oct 29 23:15:09.395178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.394680    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r" event={"ID":"7e3bda7f-64d7-41bb-9649-bba9983a995a","Type":"ContainerDied","Data":"51cd5fe616b71f364da75a916eaf910960a54c015a9ec909f130e555f92b7843"}
Oct 29 23:15:09.395178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.394713    4827 scope.go:122] "RemoveContainer" containerID="fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8"
Oct 29 23:15:09.410614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.410589    4827 scope.go:122] "RemoveContainer" containerID="fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8"
Oct 29 23:15:09.410983 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:09.410961    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8\": container with ID starting with fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8 not found: ID does not exist" containerID="fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8"
Oct 29 23:15:09.411090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.410992    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8"} err="failed to get container status \"fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8\": rpc error: code = NotFound desc = could not find container \"fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8\": container with ID starting with fdb280c7c2aedd0f8319b34f82699f0417e0e0f16f0d71b0b48404262573c4a8 not found: ID does not exist"
Oct 29 23:15:09.416502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.416485    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:09.451518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.451489    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"]
Oct 29 23:15:09.456099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.456070    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-cnn2r"]
Oct 29 23:15:09.542142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.541253    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 29 23:15:09.542142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.541308    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 29 23:15:09.751863 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:09.751834    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823210ce_20d4_48dc_99a3_1da8d708c6e7.slice/crio-1e603b8c5fe9778076800f33efec0a866f5527019ab58f76df1e6cf145f9c94f WatchSource:0}: Error finding container 1e603b8c5fe9778076800f33efec0a866f5527019ab58f76df1e6cf145f9c94f: Status 404 returned error can't find the container with id 1e603b8c5fe9778076800f33efec0a866f5527019ab58f76df1e6cf145f9c94f
Oct 29 23:15:09.754348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:09.754322    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"]
Oct 29 23:15:10.407806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:10.407749    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb" event={"ID":"823210ce-20d4-48dc-99a3-1da8d708c6e7","Type":"ContainerStarted","Data":"1e603b8c5fe9778076800f33efec0a866f5527019ab58f76df1e6cf145f9c94f"}
Oct 29 23:15:10.499037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:10.498963    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3bda7f-64d7-41bb-9649-bba9983a995a" path="/var/lib/kubelet/pods/7e3bda7f-64d7-41bb-9649-bba9983a995a/volumes"
Oct 29 23:15:10.540967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:10.540932    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 29 23:15:10.541123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:10.540996    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 29 23:15:11.303347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:11.303311    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"]
Oct 29 23:15:11.303601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:11.303556    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy" containerID="cri-o://4ee5d9df1dbe5e1e6b27049a361fdf6efe2c96dbd6ab065f6019c9d77a512579" gracePeriod=30
Oct 29 23:15:11.418441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:11.418405    4827 generic.go:356] "Generic (PLEG): container finished" podID="823210ce-20d4-48dc-99a3-1da8d708c6e7" containerID="6660836bafa8a7981ef7021d407a148af0ee954d9521bef90cfe30901e408ee1" exitCode=0
Oct 29 23:15:11.418923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:11.418458    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb" event={"ID":"823210ce-20d4-48dc-99a3-1da8d708c6e7","Type":"ContainerDied","Data":"6660836bafa8a7981ef7021d407a148af0ee954d9521bef90cfe30901e408ee1"}
Oct 29 23:15:11.559843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:11.556443    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 29 23:15:11.559843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:11.556510    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 29 23:15:12.548233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.548200    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 29 23:15:12.548714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.548263    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 29 23:15:12.625557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.625528    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:12.659530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.659500    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823210ce-20d4-48dc-99a3-1da8d708c6e7-kube-api-access-mt8m2\" (UniqueName: \"kubernetes.io/projected/823210ce-20d4-48dc-99a3-1da8d708c6e7-kube-api-access-mt8m2\") pod \"823210ce-20d4-48dc-99a3-1da8d708c6e7\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") "
Oct 29 23:15:12.659901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.659786    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-openshift-ca\") pod \"823210ce-20d4-48dc-99a3-1da8d708c6e7\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") "
Oct 29 23:15:12.659901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.659827    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-xks-ca\") pod \"823210ce-20d4-48dc-99a3-1da8d708c6e7\" (UID: \"823210ce-20d4-48dc-99a3-1da8d708c6e7\") "
Oct 29 23:15:12.660475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.660325    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-openshift-ca" pod "823210ce-20d4-48dc-99a3-1da8d708c6e7" (UID: "823210ce-20d4-48dc-99a3-1da8d708c6e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:12.660920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.660894    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:12.662201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.662173    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823210ce-20d4-48dc-99a3-1da8d708c6e7-kube-api-access-mt8m2" pod "823210ce-20d4-48dc-99a3-1da8d708c6e7" (UID: "823210ce-20d4-48dc-99a3-1da8d708c6e7"). InnerVolumeSpecName "kube-api-access-mt8m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:12.662327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.662307    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-xks-ca" pod "823210ce-20d4-48dc-99a3-1da8d708c6e7" (UID: "823210ce-20d4-48dc-99a3-1da8d708c6e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:15:12.763101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.763011    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823210ce-20d4-48dc-99a3-1da8d708c6e7-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:12.763101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:12.763037    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt8m2\" (UniqueName: \"kubernetes.io/projected/823210ce-20d4-48dc-99a3-1da8d708c6e7-kube-api-access-mt8m2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:13.441361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:13.441333    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"
Oct 29 23:15:13.441533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:13.441331    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb" event={"ID":"823210ce-20d4-48dc-99a3-1da8d708c6e7","Type":"ContainerDied","Data":"1e603b8c5fe9778076800f33efec0a866f5527019ab58f76df1e6cf145f9c94f"}
Oct 29 23:15:13.441533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:13.441489    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e603b8c5fe9778076800f33efec0a866f5527019ab58f76df1e6cf145f9c94f"
Oct 29 23:15:13.545857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:13.545815    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:15:13.546386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:13.546362    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:15:13.547336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:13.547316    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:15:14.282190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:14.282157    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:15:15.772535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:15.772499    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"]
Oct 29 23:15:15.773024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:15.772707    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" containerID="cri-o://7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284" gracePeriod=30
Oct 29 23:15:15.786069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:15.786032    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"]
Oct 29 23:15:15.786280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:15.786254    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" containerID="cri-o://8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d" gracePeriod=30
Oct 29 23:15:16.476699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:16.476668    4827 generic.go:356] "Generic (PLEG): container finished" podID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerID="4ee5d9df1dbe5e1e6b27049a361fdf6efe2c96dbd6ab065f6019c9d77a512579" exitCode=0
Oct 29 23:15:16.476928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:16.476748    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" event={"ID":"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43","Type":"ContainerDied","Data":"4ee5d9df1dbe5e1e6b27049a361fdf6efe2c96dbd6ab065f6019c9d77a512579"}
Oct 29 23:15:16.481507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:16.481484    4827 scope.go:122] "RemoveContainer" containerID="175e2ad5cf6f260bbd081cac416b338575ad276bfc10722c5f3e229504c8a71a"
Oct 29 23:15:16.482057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:16.482036    4827 scope.go:122] "RemoveContainer" containerID="c574b9198258a13335fade6b1e4aeae2636d9eab00d8ca27697a1dd47d9ff90e"
Oct 29 23:15:16.881274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:16.881238    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 23:15:17.064148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064111    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-data\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.064304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064173    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-certs\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.064363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064307    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-token\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-token\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.064483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064459    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-envoy\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.064589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064573    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-socket\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064701    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-certs" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064818    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istiod-ca-cert\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064875    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-podinfo\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.064913    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-kube-api-access-v796z\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-kube-api-access-v796z\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.065001    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-socket" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.065239    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-credential-socket\") pod \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\" (UID: \"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43\") "
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.065336    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istiod-ca-cert" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:17.065746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.065675    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-credential-socket" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:17.066312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.065843    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-data" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:17.066312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.066260    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.066312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.066285    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.066312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.066303    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.066312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.066316    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.066584 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.066329    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.068910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.068884    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-envoy" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:17.069039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.068958    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-token" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:17.069039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.068994    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-kube-api-access-v796z" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "kube-api-access-v796z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:17.069181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.069160    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-podinfo" pod "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" (UID: "a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:17.167754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.167679    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.167754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.167703    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.167754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.167713    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v796z\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-kube-api-access-v796z\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.167754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.167722    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.481352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.481279    4827 scope.go:122] "RemoveContainer" containerID="aaa5a8f98180117593ae165637b05077b39418b6d305a8a4f820a436c72ba062"
Oct 29 23:15:17.487846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.487807    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerStarted","Data":"f548de27808a1968b9ff395be089bc3c86770a1ccc558fd2c2afb01b930ba49e"}
Oct 29 23:15:17.489816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.489793    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" event={"ID":"2fdc1e99-60da-4a64-bceb-22bab429e457","Type":"ContainerStarted","Data":"bf1e4c4da1db9832bae3b1ccc7435b36feb9ab18ef0c0dc9f996a0067166c966"}
Oct 29 23:15:17.491405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.491386    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"
Oct 29 23:15:17.491512 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.491404    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86" event={"ID":"a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43","Type":"ContainerDied","Data":"e4935693193890dfcdd21ebe66e0c2d57914147d8f28adecfc372b66d4590308"}
Oct 29 23:15:17.491512 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.491454    4827 scope.go:122] "RemoveContainer" containerID="4ee5d9df1dbe5e1e6b27049a361fdf6efe2c96dbd6ab065f6019c9d77a512579"
Oct 29 23:15:17.622865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.622833    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"]
Oct 29 23:15:17.628406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:17.628380    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-xvg86"]
Oct 29 23:15:18.481500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.481475    4827 scope.go:122] "RemoveContainer" containerID="a5ff390f6abc4f0d6b694d5c7ad8c624b9debcdfcc656981f725c7cb95e3c228"
Oct 29 23:15:18.491696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.491659    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" path="/var/lib/kubelet/pods/a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43/volumes"
Oct 29 23:15:18.506235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.506204    4827 generic.go:356] "Generic (PLEG): container finished" podID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerID="f548de27808a1968b9ff395be089bc3c86770a1ccc558fd2c2afb01b930ba49e" exitCode=60
Oct 29 23:15:18.506684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.506296    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerDied","Data":"f548de27808a1968b9ff395be089bc3c86770a1ccc558fd2c2afb01b930ba49e"}
Oct 29 23:15:18.506684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.506392    4827 scope.go:122] "RemoveContainer" containerID="c574b9198258a13335fade6b1e4aeae2636d9eab00d8ca27697a1dd47d9ff90e"
Oct 29 23:15:18.506905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.506752    4827 scope.go:122] "RemoveContainer" containerID="f548de27808a1968b9ff395be089bc3c86770a1ccc558fd2c2afb01b930ba49e"
Oct 29 23:15:18.507150 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:18.507117    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq_odh-ai-gateway-infra(a78ac709-b2ed-447e-94e8-c5c60e2df3a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4"
Oct 29 23:15:18.513600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.513575    4827 generic.go:356] "Generic (PLEG): container finished" podID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerID="bf1e4c4da1db9832bae3b1ccc7435b36feb9ab18ef0c0dc9f996a0067166c966" exitCode=60
Oct 29 23:15:18.513726 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.513649    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" event={"ID":"2fdc1e99-60da-4a64-bceb-22bab429e457","Type":"ContainerDied","Data":"bf1e4c4da1db9832bae3b1ccc7435b36feb9ab18ef0c0dc9f996a0067166c966"}
Oct 29 23:15:18.514155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.514132    4827 scope.go:122] "RemoveContainer" containerID="bf1e4c4da1db9832bae3b1ccc7435b36feb9ab18ef0c0dc9f996a0067166c966"
Oct 29 23:15:18.514483 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:18.514462    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z_odh-ai-gateway-infra(2fdc1e99-60da-4a64-bceb-22bab429e457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457"
Oct 29 23:15:18.516163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.516144    4827 generic.go:356] "Generic (PLEG): container finished" podID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerID="c00bcb8109c4877c16a414272c1ebeac56b24fc1890307a20f45dcff341c858f" exitCode=60
Oct 29 23:15:18.516240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.516178    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" event={"ID":"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91","Type":"ContainerDied","Data":"c00bcb8109c4877c16a414272c1ebeac56b24fc1890307a20f45dcff341c858f"}
Oct 29 23:15:18.516597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.516577    4827 scope.go:122] "RemoveContainer" containerID="c00bcb8109c4877c16a414272c1ebeac56b24fc1890307a20f45dcff341c858f"
Oct 29 23:15:18.516999 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:18.516928    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn_odh-ai-gateway-infra(4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91"
Oct 29 23:15:18.526852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.526833    4827 scope.go:122] "RemoveContainer" containerID="175e2ad5cf6f260bbd081cac416b338575ad276bfc10722c5f3e229504c8a71a"
Oct 29 23:15:18.540792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.540755    4827 scope.go:122] "RemoveContainer" containerID="aaa5a8f98180117593ae165637b05077b39418b6d305a8a4f820a436c72ba062"
Oct 29 23:15:18.544867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.544845    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"]
Oct 29 23:15:18.595279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:18.595250    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"]
Oct 29 23:15:19.480669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.480641    4827 scope.go:122] "RemoveContainer" containerID="832539c383e13ec094c16d9495ef53bf27851bc2b1e13a28e431060338228574"
Oct 29 23:15:19.480890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.480874    4827 scope.go:122] "RemoveContainer" containerID="098d61ce375acb4d062de4d901e29a3b6419ff989384ec0c26c614f0321d52f6"
Oct 29 23:15:19.539981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.539946    4827 generic.go:356] "Generic (PLEG): container finished" podID="38d21edd-2777-4885-b045-3c012d00e931" containerID="b523859848f67bb6ec5e418fbd1dd32be3fc72bbdcecdffb1933e7b2419a08cc" exitCode=60
Oct 29 23:15:19.540385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.540125    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" event={"ID":"38d21edd-2777-4885-b045-3c012d00e931","Type":"ContainerDied","Data":"b523859848f67bb6ec5e418fbd1dd32be3fc72bbdcecdffb1933e7b2419a08cc"}
Oct 29 23:15:19.540385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.540196    4827 scope.go:122] "RemoveContainer" containerID="a5ff390f6abc4f0d6b694d5c7ad8c624b9debcdfcc656981f725c7cb95e3c228"
Oct 29 23:15:19.541329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.541221    4827 scope.go:122] "RemoveContainer" containerID="b523859848f67bb6ec5e418fbd1dd32be3fc72bbdcecdffb1933e7b2419a08cc"
Oct 29 23:15:19.541700 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:19.541678    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2_odh-ai-gateway-infra(38d21edd-2777-4885-b045-3c012d00e931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" podUID="38d21edd-2777-4885-b045-3c012d00e931"
Oct 29 23:15:19.706063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.705272    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"]
Oct 29 23:15:19.853505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.853153    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:19.883906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.883834    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:19.928018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.927936    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fdc1e99-60da-4a64-bceb-22bab429e457-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc1e99-60da-4a64-bceb-22bab429e457-maas-api-service-ca\") pod \"2fdc1e99-60da-4a64-bceb-22bab429e457\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") "
Oct 29 23:15:19.928181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.928041    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fdc1e99-60da-4a64-bceb-22bab429e457-kube-api-access-6nqdf\" (UniqueName: \"kubernetes.io/projected/2fdc1e99-60da-4a64-bceb-22bab429e457-kube-api-access-6nqdf\") pod \"2fdc1e99-60da-4a64-bceb-22bab429e457\" (UID: \"2fdc1e99-60da-4a64-bceb-22bab429e457\") "
Oct 29 23:15:19.928251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.928228    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-maas-api-service-ca\") pod \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") "
Oct 29 23:15:19.928338 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.928311    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdc1e99-60da-4a64-bceb-22bab429e457-maas-api-service-ca" pod "2fdc1e99-60da-4a64-bceb-22bab429e457" (UID: "2fdc1e99-60da-4a64-bceb-22bab429e457"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:19.928392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.928327    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-kube-api-access-67s4q\" (UniqueName: \"kubernetes.io/projected/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-kube-api-access-67s4q\") pod \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\" (UID: \"a78ac709-b2ed-447e-94e8-c5c60e2df3a4\") "
Oct 29 23:15:19.928615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.928592    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-maas-api-service-ca" pod "a78ac709-b2ed-447e-94e8-c5c60e2df3a4" (UID: "a78ac709-b2ed-447e-94e8-c5c60e2df3a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:19.929248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.929204    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:19.929389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.929254    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc1e99-60da-4a64-bceb-22bab429e457-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:19.965875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.965840    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-kube-api-access-67s4q" pod "a78ac709-b2ed-447e-94e8-c5c60e2df3a4" (UID: "a78ac709-b2ed-447e-94e8-c5c60e2df3a4"). InnerVolumeSpecName "kube-api-access-67s4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:19.966023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.965917    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdc1e99-60da-4a64-bceb-22bab429e457-kube-api-access-6nqdf" pod "2fdc1e99-60da-4a64-bceb-22bab429e457" (UID: "2fdc1e99-60da-4a64-bceb-22bab429e457"). InnerVolumeSpecName "kube-api-access-6nqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:19.975964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:19.975942    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:20.030745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.030703    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-maas-api-service-ca\") pod \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") "
Oct 29 23:15:20.031012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.030884    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-kube-api-access-zfpj2\" (UniqueName: \"kubernetes.io/projected/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-kube-api-access-zfpj2\") pod \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\" (UID: \"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91\") "
Oct 29 23:15:20.031266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.031170    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-maas-api-service-ca" pod "4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" (UID: "4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:20.031657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.031637    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.031750 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.031662    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67s4q\" (UniqueName: \"kubernetes.io/projected/a78ac709-b2ed-447e-94e8-c5c60e2df3a4-kube-api-access-67s4q\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.031750 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.031676    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nqdf\" (UniqueName: \"kubernetes.io/projected/2fdc1e99-60da-4a64-bceb-22bab429e457-kube-api-access-6nqdf\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.033342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.033318    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-kube-api-access-zfpj2" pod "4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" (UID: "4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91"). InnerVolumeSpecName "kube-api-access-zfpj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:20.132963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.132930    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfpj2\" (UniqueName: \"kubernetes.io/projected/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91-kube-api-access-zfpj2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.238555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.238467    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-4rrmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:15:20.238823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.238753    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:15:20.319127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.319093    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnpsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:15:20.319313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.319154    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:15:20.550209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.550175    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"
Oct 29 23:15:20.550645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.550218    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq" event={"ID":"a78ac709-b2ed-447e-94e8-c5c60e2df3a4","Type":"ContainerDied","Data":"0deea2e9f26784358c939c1d0b5e7036ec974d32a2429506d0199a9ca63eb0c3"}
Oct 29 23:15:20.550645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.550280    4827 scope.go:122] "RemoveContainer" containerID="f548de27808a1968b9ff395be089bc3c86770a1ccc558fd2c2afb01b930ba49e"
Oct 29 23:15:20.551749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.551732    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"
Oct 29 23:15:20.551852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.551762    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z" event={"ID":"2fdc1e99-60da-4a64-bceb-22bab429e457","Type":"ContainerDied","Data":"28c62f2f3a4f13fb5cb5e26b6c60020fd32984edb0ae1a8acb5c14b799f0a9ca"}
Oct 29 23:15:20.553601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.553575    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"
Oct 29 23:15:20.553710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.553597    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn" event={"ID":"4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91","Type":"ContainerDied","Data":"42a378fe79b4112fbf75594e4870db78145c7834e30e28762ec992a459a3ef27"}
Oct 29 23:15:20.558085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.558063    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerStarted","Data":"546b9b156a2684144a141b7352547eb8358c07214451024a54929347856b4cd1"}
Oct 29 23:15:20.559993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.559969    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" event={"ID":"9bccacf6-8e28-4367-88c3-48382b468dda","Type":"ContainerStarted","Data":"fdf7f5dee99ad1fbffa8b3b18eb709319ad0e05d5f3e51c3d8a170d757520414"}
Oct 29 23:15:20.564194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.563926    4827 scope.go:122] "RemoveContainer" containerID="bf1e4c4da1db9832bae3b1ccc7435b36feb9ab18ef0c0dc9f996a0067166c966"
Oct 29 23:15:20.575781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.575749    4827 scope.go:122] "RemoveContainer" containerID="c00bcb8109c4877c16a414272c1ebeac56b24fc1890307a20f45dcff341c858f"
Oct 29 23:15:20.608307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.608267    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"]
Oct 29 23:15:20.613082 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.613060    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5e52e0-29888595-whbfq"]
Oct 29 23:15:20.633437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.633412    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"]
Oct 29 23:15:20.697648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.697611    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"]
Oct 29 23:15:20.702832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.702808    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-32717a-29888595-6pg4z"]
Oct 29 23:15:20.784861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.779669    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"]
Oct 29 23:15:20.785049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.784939    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888595-mwksn"]
Oct 29 23:15:20.942393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:20.942365    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:21.052532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.052500    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d21edd-2777-4885-b045-3c012d00e931-kube-api-access-8z2mq\" (UniqueName: \"kubernetes.io/projected/38d21edd-2777-4885-b045-3c012d00e931-kube-api-access-8z2mq\") pod \"38d21edd-2777-4885-b045-3c012d00e931\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") "
Oct 29 23:15:21.054103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.053225    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d21edd-2777-4885-b045-3c012d00e931-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d21edd-2777-4885-b045-3c012d00e931-maas-api-service-ca\") pod \"38d21edd-2777-4885-b045-3c012d00e931\" (UID: \"38d21edd-2777-4885-b045-3c012d00e931\") "
Oct 29 23:15:21.054620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.054593    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d21edd-2777-4885-b045-3c012d00e931-maas-api-service-ca" pod "38d21edd-2777-4885-b045-3c012d00e931" (UID: "38d21edd-2777-4885-b045-3c012d00e931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:21.054958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.054919    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d21edd-2777-4885-b045-3c012d00e931-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.057678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.057653    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d21edd-2777-4885-b045-3c012d00e931-kube-api-access-8z2mq" pod "38d21edd-2777-4885-b045-3c012d00e931" (UID: "38d21edd-2777-4885-b045-3c012d00e931"). InnerVolumeSpecName "kube-api-access-8z2mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:21.113564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.113536    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 23:15:21.144915 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.144895    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 23:15:21.155717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.155696    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-envoy\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.155882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.155753    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-kube-api-access-hnlrt\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-kube-api-access-hnlrt\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.155882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.155856    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-token\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-token\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.156036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.155896    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istiod-ca-cert\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.156036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.155941    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-credential-socket\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.156036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.155978    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-certs\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.156203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156069    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-podinfo\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.156203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156190    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-data\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.156328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156254    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-credential-socket" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.156328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156287    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-certs" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.156328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156301    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-socket\") pod \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\" (UID: \"f3419fb5-2606-4c5e-89fd-7efa2ab90437\") "
Oct 29 23:15:21.156485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156328    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istiod-ca-cert" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:21.156550 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156511    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-data" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.156708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156681    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-socket" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.157008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.156989    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z2mq\" (UniqueName: \"kubernetes.io/projected/38d21edd-2777-4885-b045-3c012d00e931-kube-api-access-8z2mq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.157093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.157010    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.157093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.157019    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.157093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.157029    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.157093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.157038    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.157093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.157045    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.158017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.157998    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-kube-api-access-hnlrt" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "kube-api-access-hnlrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:21.158116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.158086    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-envoy" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.158379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.158365    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-token" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:21.158417 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.158403    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-podinfo" pod "f3419fb5-2606-4c5e-89fd-7efa2ab90437" (UID: "f3419fb5-2606-4c5e-89fd-7efa2ab90437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:21.257662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.257583    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-credential-socket\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.257903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.257673    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-socket\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.257903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.257739    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-data\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.257903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.257890    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3debd53f-0d24-45f7-adae-dd8331f42ebb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3debd53f-0d24-45f7-adae-dd8331f42ebb-istiod-ca-cert\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.258100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.257924    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-kube-api-access-8g7b7\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-kube-api-access-8g7b7\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.258100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.257969    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-certs\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.258100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.258065    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-podinfo\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.258317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.258117    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-credential-socket" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.258453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.258220    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-data" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.258453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.258267    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-envoy\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.258453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.258406    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-token\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-token\") pod \"3debd53f-0d24-45f7-adae-dd8331f42ebb\" (UID: \"3debd53f-0d24-45f7-adae-dd8331f42ebb\") "
Oct 29 23:15:21.259026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.258995    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-certs" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.259207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259121    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-socket" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.259365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259330    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259353    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlrt\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-kube-api-access-hnlrt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259367    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259380    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259392    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259406    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3419fb5-2606-4c5e-89fd-7efa2ab90437-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259419    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259433    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.259747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.259492    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3debd53f-0d24-45f7-adae-dd8331f42ebb-istiod-ca-cert" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:21.261363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.261329    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-envoy" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:21.261585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.261565    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-kube-api-access-8g7b7" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "kube-api-access-8g7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:21.261674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.261637    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-token" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:21.261913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.261895    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-podinfo" pod "3debd53f-0d24-45f7-adae-dd8331f42ebb" (UID: "3debd53f-0d24-45f7-adae-dd8331f42ebb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:21.361352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.361273    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3debd53f-0d24-45f7-adae-dd8331f42ebb-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.361352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.361302    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7b7\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-kube-api-access-8g7b7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.361352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.361312    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.361352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.361322    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.361352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.361331    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3debd53f-0d24-45f7-adae-dd8331f42ebb-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.574614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.574580    4827 generic.go:356] "Generic (PLEG): container finished" podID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerID="8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d" exitCode=0
Oct 29 23:15:21.575126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.574734    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"
Oct 29 23:15:21.575126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.574859    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" event={"ID":"3debd53f-0d24-45f7-adae-dd8331f42ebb","Type":"ContainerDied","Data":"8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d"}
Oct 29 23:15:21.575126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.574941    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr" event={"ID":"3debd53f-0d24-45f7-adae-dd8331f42ebb","Type":"ContainerDied","Data":"1e18dddef368a1741416fe946c58c8f500b6c7fdc6c67eea3b23686339220abd"}
Oct 29 23:15:21.575126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.574977    4827 scope.go:122] "RemoveContainer" containerID="8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d"
Oct 29 23:15:21.583861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.583833    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2" event={"ID":"38d21edd-2777-4885-b045-3c012d00e931","Type":"ContainerDied","Data":"eba27cd7b058d9e13354e75d8b488287dd7972bde20140a58aedd65ac794b983"}
Oct 29 23:15:21.584015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.583924    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"
Oct 29 23:15:21.590293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.590270    4827 generic.go:356] "Generic (PLEG): container finished" podID="4256f4cf-13ca-4286-91a8-485455a8694d" containerID="546b9b156a2684144a141b7352547eb8358c07214451024a54929347856b4cd1" exitCode=60
Oct 29 23:15:21.591572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.590491    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerDied","Data":"546b9b156a2684144a141b7352547eb8358c07214451024a54929347856b4cd1"}
Oct 29 23:15:21.591572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.591109    4827 scope.go:122] "RemoveContainer" containerID="546b9b156a2684144a141b7352547eb8358c07214451024a54929347856b4cd1"
Oct 29 23:15:21.591572 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:21.591516    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-sx7sd_odh-ai-gateway-infra(4256f4cf-13ca-4286-91a8-485455a8694d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" podUID="4256f4cf-13ca-4286-91a8-485455a8694d"
Oct 29 23:15:21.592988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.592970    4827 scope.go:122] "RemoveContainer" containerID="8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d"
Oct 29 23:15:21.593274 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:21.593247    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d\": container with ID starting with 8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d not found: ID does not exist" containerID="8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d"
Oct 29 23:15:21.593379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.593284    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d"} err="failed to get container status \"8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d\": rpc error: code = NotFound desc = could not find container \"8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d\": container with ID starting with 8143aea74c30f13bf0264100a081499d27cd24611c29b2c82e3c713469a6589d not found: ID does not exist"
Oct 29 23:15:21.593379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.593308    4827 scope.go:122] "RemoveContainer" containerID="b523859848f67bb6ec5e418fbd1dd32be3fc72bbdcecdffb1933e7b2419a08cc"
Oct 29 23:15:21.594818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.594745    4827 generic.go:356] "Generic (PLEG): container finished" podID="9bccacf6-8e28-4367-88c3-48382b468dda" containerID="fdf7f5dee99ad1fbffa8b3b18eb709319ad0e05d5f3e51c3d8a170d757520414" exitCode=60
Oct 29 23:15:21.594902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.594858    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" event={"ID":"9bccacf6-8e28-4367-88c3-48382b468dda","Type":"ContainerDied","Data":"fdf7f5dee99ad1fbffa8b3b18eb709319ad0e05d5f3e51c3d8a170d757520414"}
Oct 29 23:15:21.595316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.595294    4827 scope.go:122] "RemoveContainer" containerID="fdf7f5dee99ad1fbffa8b3b18eb709319ad0e05d5f3e51c3d8a170d757520414"
Oct 29 23:15:21.595703 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:21.595671    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s_odh-ai-gateway-infra(9bccacf6-8e28-4367-88c3-48382b468dda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" podUID="9bccacf6-8e28-4367-88c3-48382b468dda"
Oct 29 23:15:21.597904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.597879    4827 generic.go:356] "Generic (PLEG): container finished" podID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerID="7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284" exitCode=0
Oct 29 23:15:21.597998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.597938    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" event={"ID":"f3419fb5-2606-4c5e-89fd-7efa2ab90437","Type":"ContainerDied","Data":"7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284"}
Oct 29 23:15:21.597998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.597975    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt" event={"ID":"f3419fb5-2606-4c5e-89fd-7efa2ab90437","Type":"ContainerDied","Data":"36c900f656765268fd514fd3f5bda444b858102fb81691064d48d2627ab3addf"}
Oct 29 23:15:21.597998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.597981    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"
Oct 29 23:15:21.607810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.607791    4827 scope.go:122] "RemoveContainer" containerID="832539c383e13ec094c16d9495ef53bf27851bc2b1e13a28e431060338228574"
Oct 29 23:15:21.625932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.625868    4827 scope.go:122] "RemoveContainer" containerID="098d61ce375acb4d062de4d901e29a3b6419ff989384ec0c26c614f0321d52f6"
Oct 29 23:15:21.642462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.642440    4827 scope.go:122] "RemoveContainer" containerID="7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284"
Oct 29 23:15:21.653537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.653508    4827 scope.go:122] "RemoveContainer" containerID="7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284"
Oct 29 23:15:21.653860 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:21.653839    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284\": container with ID starting with 7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284 not found: ID does not exist" containerID="7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284"
Oct 29 23:15:21.653960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.653871    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284"} err="failed to get container status \"7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284\": rpc error: code = NotFound desc = could not find container \"7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284\": container with ID starting with 7c307fd99e5e78cfe7addeed600287002f7e891aa53e5e85af42de2215e59284 not found: ID does not exist"
Oct 29 23:15:21.670557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.670527    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"]
Oct 29 23:15:21.707814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.707745    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"]
Oct 29 23:15:21.712316 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.712290    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnpsr"]
Oct 29 23:15:21.740864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.740833    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"]
Oct 29 23:15:21.746305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.746280    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-4rrmt"]
Oct 29 23:15:21.750560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.750538    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"]
Oct 29 23:15:21.783455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.783422    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"]
Oct 29 23:15:21.803341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:21.803308    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888595-lqfk2"]
Oct 29 23:15:22.550800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.545268    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" path="/var/lib/kubelet/pods/2fdc1e99-60da-4a64-bceb-22bab429e457/volumes"
Oct 29 23:15:22.552000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.551969    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d21edd-2777-4885-b045-3c012d00e931" path="/var/lib/kubelet/pods/38d21edd-2777-4885-b045-3c012d00e931/volumes"
Oct 29 23:15:22.561797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.553049    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" path="/var/lib/kubelet/pods/3debd53f-0d24-45f7-adae-dd8331f42ebb/volumes"
Oct 29 23:15:22.562995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.562965    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" path="/var/lib/kubelet/pods/4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91/volumes"
Oct 29 23:15:22.564069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.564048    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" path="/var/lib/kubelet/pods/a78ac709-b2ed-447e-94e8-c5c60e2df3a4/volumes"
Oct 29 23:15:22.569428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.569405    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" path="/var/lib/kubelet/pods/f3419fb5-2606-4c5e-89fd-7efa2ab90437/volumes"
Oct 29 23:15:22.904364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.904337    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:22.947796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.947723    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:22.979784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.979742    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bccacf6-8e28-4367-88c3-48382b468dda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccacf6-8e28-4367-88c3-48382b468dda-maas-api-service-ca\") pod \"9bccacf6-8e28-4367-88c3-48382b468dda\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") "
Oct 29 23:15:22.980165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.980136    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bccacf6-8e28-4367-88c3-48382b468dda-maas-api-service-ca" pod "9bccacf6-8e28-4367-88c3-48382b468dda" (UID: "9bccacf6-8e28-4367-88c3-48382b468dda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:22.980285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.980223    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bccacf6-8e28-4367-88c3-48382b468dda-kube-api-access-x9xm7\" (UniqueName: \"kubernetes.io/projected/9bccacf6-8e28-4367-88c3-48382b468dda-kube-api-access-x9xm7\") pod \"9bccacf6-8e28-4367-88c3-48382b468dda\" (UID: \"9bccacf6-8e28-4367-88c3-48382b468dda\") "
Oct 29 23:15:22.981135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.981112    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccacf6-8e28-4367-88c3-48382b468dda-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:22.982516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:22.982492    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bccacf6-8e28-4367-88c3-48382b468dda-kube-api-access-x9xm7" pod "9bccacf6-8e28-4367-88c3-48382b468dda" (UID: "9bccacf6-8e28-4367-88c3-48382b468dda"). InnerVolumeSpecName "kube-api-access-x9xm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:23.082241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.082153    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4256f4cf-13ca-4286-91a8-485455a8694d-kube-api-access-87ztj\" (UniqueName: \"kubernetes.io/projected/4256f4cf-13ca-4286-91a8-485455a8694d-kube-api-access-87ztj\") pod \"4256f4cf-13ca-4286-91a8-485455a8694d\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") "
Oct 29 23:15:23.082424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.082265    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4256f4cf-13ca-4286-91a8-485455a8694d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4256f4cf-13ca-4286-91a8-485455a8694d-maas-api-service-ca\") pod \"4256f4cf-13ca-4286-91a8-485455a8694d\" (UID: \"4256f4cf-13ca-4286-91a8-485455a8694d\") "
Oct 29 23:15:23.083252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.083221    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9xm7\" (UniqueName: \"kubernetes.io/projected/9bccacf6-8e28-4367-88c3-48382b468dda-kube-api-access-x9xm7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:23.084087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.084058    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4256f4cf-13ca-4286-91a8-485455a8694d-maas-api-service-ca" pod "4256f4cf-13ca-4286-91a8-485455a8694d" (UID: "4256f4cf-13ca-4286-91a8-485455a8694d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:23.086040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.085990    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4256f4cf-13ca-4286-91a8-485455a8694d-kube-api-access-87ztj" pod "4256f4cf-13ca-4286-91a8-485455a8694d" (UID: "4256f4cf-13ca-4286-91a8-485455a8694d"). InnerVolumeSpecName "kube-api-access-87ztj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:23.184590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.184557    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4256f4cf-13ca-4286-91a8-485455a8694d-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:23.184590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.184585    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ztj\" (UniqueName: \"kubernetes.io/projected/4256f4cf-13ca-4286-91a8-485455a8694d-kube-api-access-87ztj\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:23.691373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.691341    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"
Oct 29 23:15:23.691373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.691367    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd" event={"ID":"4256f4cf-13ca-4286-91a8-485455a8694d","Type":"ContainerDied","Data":"cfa5332cec5d66782e39b9b583ecd3f86866837a556bbf011340a538c119a0e5"}
Oct 29 23:15:23.691605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.691422    4827 scope.go:122] "RemoveContainer" containerID="546b9b156a2684144a141b7352547eb8358c07214451024a54929347856b4cd1"
Oct 29 23:15:23.693029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.693009    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"
Oct 29 23:15:23.693157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.693001    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s" event={"ID":"9bccacf6-8e28-4367-88c3-48382b468dda","Type":"ContainerDied","Data":"12f382abb54b22918805edb01c5a9cf1c6e9c9c7e44d60485c9d0c5d8e05ee26"}
Oct 29 23:15:23.705756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.705741    4827 scope.go:122] "RemoveContainer" containerID="fdf7f5dee99ad1fbffa8b3b18eb709319ad0e05d5f3e51c3d8a170d757520414"
Oct 29 23:15:23.746258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.746231    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"]
Oct 29 23:15:23.757514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.757489    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-sx7sd"]
Oct 29 23:15:23.785441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.785417    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"]
Oct 29 23:15:23.789449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:23.789427    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-31226c-29888595-vfk2s"]
Oct 29 23:15:24.151461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.151426    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"]
Oct 29 23:15:24.152616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152592    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:24.152616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152615    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152636    4827 cpu_manager.go:414] "containerMap: removing container" podUID="823210ce-20d4-48dc-99a3-1da8d708c6e7" containerName="revoke-keys"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152644    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823210ce-20d4-48dc-99a3-1da8d708c6e7" containerName="revoke-keys"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152667    4827 cpu_manager.go:414] "containerMap: removing container" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152675    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152687    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152694    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152713    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152721    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152733    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152741    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152757    4827 cpu_manager.go:414] "containerMap: removing container" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152778    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152789    4827 cpu_manager.go:414] "containerMap: removing container" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152795    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152807    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152815    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152836    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152844    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.152852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152861    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152869    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152885    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152893    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152910    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152917    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152931    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152938    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152949    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152956    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152971    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152978    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.152997    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153004    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153022    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153028    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153041    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153047    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153246    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="823210ce-20d4-48dc-99a3-1da8d708c6e7" containerName="revoke-keys"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153261    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153270    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153282    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153293    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153304    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153314    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d21edd-2777-4885-b045-3c012d00e931" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153323    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3419fb5-2606-4c5e-89fd-7efa2ab90437" containerName="istio-proxy"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153336    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153345    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153355    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153364    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153375    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153383    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153393    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79d5ee2-d16c-4cf4-9fdb-b3b2e55c4c43" containerName="istio-proxy"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153402    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3debd53f-0d24-45f7-adae-dd8331f42ebb" containerName="istio-proxy"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153413    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153424    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153433    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:24.153810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.153442    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:24.159243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.159216    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.179720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.179697    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"]
Oct 29 23:15:24.184889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.184866    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1056cc-openshift-default-dockercfg-6p8ms\""
Oct 29 23:15:24.324807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.324381    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.324807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.324552    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.324807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.324650    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.324807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.324713    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.324807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.324805    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lzn\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-kube-api-access-n6lzn\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.325291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.325010    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.325291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.325066    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.325291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.325098    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.325291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.325157    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427091    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427194    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427265    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427329    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lzn\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-kube-api-access-n6lzn\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427515    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427579    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427610    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427660    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.427728    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.428815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.428567    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.429107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.429077    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.430161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.429384    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.448789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.438925    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.448789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.439392    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.487137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.484128    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.493442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.493395    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.494055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.493485    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.494244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.493493    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lzn\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-kube-api-access-n6lzn\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.494860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.494841    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:24.565062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.563970    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4256f4cf-13ca-4286-91a8-485455a8694d" path="/var/lib/kubelet/pods/4256f4cf-13ca-4286-91a8-485455a8694d/volumes"
Oct 29 23:15:24.565062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.564634    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bccacf6-8e28-4367-88c3-48382b468dda" path="/var/lib/kubelet/pods/9bccacf6-8e28-4367-88c3-48382b468dda/volumes"
Oct 29 23:15:24.848343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.848317    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"]
Oct 29 23:15:24.848494 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:24.848430    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80096ee0_b2d5_4fac_9a73_6a05813c0ff0.slice/crio-db2cb33a01aeed089ad706c6eb4db237e143458fbde0f3104f190a12e4679d4a WatchSource:0}: Error finding container db2cb33a01aeed089ad706c6eb4db237e143458fbde0f3104f190a12e4679d4a: Status 404 returned error can't find the container with id db2cb33a01aeed089ad706c6eb4db237e143458fbde0f3104f190a12e4679d4a
Oct 29 23:15:24.851820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.851778    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:24.851918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.851866    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:24.851918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:24.851910    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:25.803619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:25.803580    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" event={"ID":"80096ee0-b2d5-4fac-9a73-6a05813c0ff0","Type":"ContainerStarted","Data":"c694795834d6ca69cb9e0b536662a4ea9eaf4da0c7946af8b3eda05d78163185"}
Oct 29 23:15:25.804117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:25.803645    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" event={"ID":"80096ee0-b2d5-4fac-9a73-6a05813c0ff0","Type":"ContainerStarted","Data":"db2cb33a01aeed089ad706c6eb4db237e143458fbde0f3104f190a12e4679d4a"}
Oct 29 23:15:25.873935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:25.873884    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podStartSLOduration=1.8738695170000002 podStartE2EDuration="1.873869517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:25.860296534 +0000 UTC m=+1961.483207554" watchObservedRunningTime="2026-10-29 23:15:25.873869517 +0000 UTC m=+1961.496780536"
Oct 29 23:15:26.495437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:26.495407    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:26.501808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:26.501754    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:26.812409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:26.812321    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:26.813315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:26.813296    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:15:29.445791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:29.442864    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:15:29.445791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:29.443855    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:15:29.455788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:29.454857    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:15:29.477958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:29.475901    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:15:29.509027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:29.508983    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:15:30.317620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.317591    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"]
Oct 29 23:15:30.318360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318343    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:30.318415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318360    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:30.318415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318398    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:30.318415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318405    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78ac709-b2ed-447e-94e8-c5c60e2df3a4" containerName="cleanup"
Oct 29 23:15:30.318522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318426    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:30.318522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318431    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:30.318588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318545    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9e70cd-e47d-4a02-bcb0-ed86a6e46c91" containerName="cleanup"
Oct 29 23:15:30.318588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.318555    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdc1e99-60da-4a64-bceb-22bab429e457" containerName="cleanup"
Oct 29 23:15:30.322300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.322276    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:30.341008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.340966    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"]
Oct 29 23:15:30.348241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.348220    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1056cc\""
Oct 29 23:15:30.348241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.348228    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1056cc-dockercfg-tqbz7\""
Oct 29 23:15:30.411208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.411172    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdk7c\" (UniqueName: \"kubernetes.io/projected/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-kube-api-access-xdk7c\") pod \"payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:30.411465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.411435    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:30.413414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.413390    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"]
Oct 29 23:15:30.418095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.418075    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:30.420361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.420332    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"]
Oct 29 23:15:30.550798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.528089    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdk7c\" (UniqueName: \"kubernetes.io/projected/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-kube-api-access-xdk7c\") pod \"payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:30.550798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.528308    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:30.550798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.529399    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:30.685835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.681842    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdk7c\" (UniqueName: \"kubernetes.io/projected/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-kube-api-access-xdk7c\") pod \"payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:30.709416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.709369    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54chc\" (UniqueName: \"kubernetes.io/projected/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-kube-api-access-54chc\") pod \"payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:30.786788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.746040    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:30.885788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.882581    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:30.885788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.882780    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54chc\" (UniqueName: \"kubernetes.io/projected/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-kube-api-access-54chc\") pod \"payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:30.885788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.884275    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:30.926788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.921472    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54chc\" (UniqueName: \"kubernetes.io/projected/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-kube-api-access-54chc\") pod \"payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:30.954858 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:30.944095    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:31.205064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:31.204171    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:31.397528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:31.397475    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"]
Oct 29 23:15:31.398454 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:31.398428    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ca3ed2_9f7f_49ec_b1e1_94f93079a2e8.slice/crio-ab20e216979e28f012112ae2ad25252dfa777c1919bbc888886d0eb31263c4c4 WatchSource:0}: Error finding container ab20e216979e28f012112ae2ad25252dfa777c1919bbc888886d0eb31263c4c4: Status 404 returned error can't find the container with id ab20e216979e28f012112ae2ad25252dfa777c1919bbc888886d0eb31263c4c4
Oct 29 23:15:31.593534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:31.593495    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"]
Oct 29 23:15:31.595116 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:31.595089    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3921fe_1a6d_4446_9d78_1a6fcaa907b4.slice/crio-07938ba368e074f3d938b85ad6a5f71598338acfa88cbb7d9c507a45b8d31119 WatchSource:0}: Error finding container 07938ba368e074f3d938b85ad6a5f71598338acfa88cbb7d9c507a45b8d31119: Status 404 returned error can't find the container with id 07938ba368e074f3d938b85ad6a5f71598338acfa88cbb7d9c507a45b8d31119
Oct 29 23:15:32.044349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.044313    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg" event={"ID":"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8","Type":"ContainerStarted","Data":"4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423"}
Oct 29 23:15:32.044560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.044386    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:32.044560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.044414    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg" event={"ID":"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8","Type":"ContainerStarted","Data":"ab20e216979e28f012112ae2ad25252dfa777c1919bbc888886d0eb31263c4c4"}
Oct 29 23:15:32.046010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.045975    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" event={"ID":"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4","Type":"ContainerStarted","Data":"1fa51bdf753d44ad72bbe8d0accae8b578b5e8a9e4f5bf3ef4754a4b7659ba53"}
Oct 29 23:15:32.046148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.046027    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" event={"ID":"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4","Type":"ContainerStarted","Data":"07938ba368e074f3d938b85ad6a5f71598338acfa88cbb7d9c507a45b8d31119"}
Oct 29 23:15:32.046148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.046054    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:32.089791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.089166    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg" podStartSLOduration=2.08914901 podStartE2EDuration="2.08914901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:32.086718508 +0000 UTC m=+1967.709629554" watchObservedRunningTime="2026-10-29 23:15:32.08914901 +0000 UTC m=+1967.712060031"
Oct 29 23:15:32.128472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:32.128421    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" podStartSLOduration=2.128409066 podStartE2EDuration="2.128409066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:32.127594556 +0000 UTC m=+1967.750505577" watchObservedRunningTime="2026-10-29 23:15:32.128409066 +0000 UTC m=+1967.751320085"
Oct 29 23:15:32.828862 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:32.828826    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:15:38.069416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:38.069319    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"]
Oct 29 23:15:38.069980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:38.069521    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" containerID="cri-o://e97f672fd604ff3bfb1e0ad501d13fe417e11679a6c8e4e3329480561ace7b08" gracePeriod=30
Oct 29 23:15:40.217630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:40.217600    4827 generic.go:356] "Generic (PLEG): container finished" podID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerID="85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4" exitCode=0
Oct 29 23:15:40.218094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:40.217666    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" event={"ID":"ace4e306-cefd-462b-9276-812fa8fb1c7b","Type":"ContainerDied","Data":"85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4"}
Oct 29 23:15:40.218166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:40.218135    4827 scope.go:122] "RemoveContainer" containerID="85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4"
Oct 29 23:15:40.220080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:40.220052    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:40.220171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:40.220123    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:40.220216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:40.220168    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:41.227824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.227792    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" event={"ID":"ace4e306-cefd-462b-9276-812fa8fb1c7b","Type":"ContainerStarted","Data":"9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236"}
Oct 29 23:15:41.399850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.399813    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"]
Oct 29 23:15:41.403904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.403884    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.421462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.421432    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-462fb2-serving-cert\""
Oct 29 23:15:41.427979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.427953    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"]
Oct 29 23:15:41.433940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.433654    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxgt\" (UniqueName: \"kubernetes.io/projected/a94dd682-8a3b-4067-9414-3b6041b34051-kube-api-access-6cxgt\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.433940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.433709    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94dd682-8a3b-4067-9414-3b6041b34051-openshift-service-ca\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.434164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.434027    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.434231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.434160    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-metrics-certs\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.535875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.535805    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.536063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.535900    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-metrics-certs\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.536063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.535963    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxgt\" (UniqueName: \"kubernetes.io/projected/a94dd682-8a3b-4067-9414-3b6041b34051-kube-api-access-6cxgt\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.536063 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:41.535973    4827 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-serving-cert: secret "maas-api-e2e-inf-a-462fb2-serving-cert" not found
Oct 29 23:15:41.536063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.535991    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94dd682-8a3b-4067-9414-3b6041b34051-openshift-service-ca\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.536063 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:15:41.536043    4827 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls podName:a94dd682-8a3b-4067-9414-3b6041b34051 nodeName:}" failed. No retries permitted until 2026-10-29 23:15:42.036022014 +0000 UTC m=+1977.658933027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls") pod "maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd" (UID: "a94dd682-8a3b-4067-9414-3b6041b34051") : secret "maas-api-e2e-inf-a-462fb2-serving-cert" not found
Oct 29 23:15:41.536671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.536644    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94dd682-8a3b-4067-9414-3b6041b34051-openshift-service-ca\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.538390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.538370    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-metrics-certs\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:41.561878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:41.561856    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxgt\" (UniqueName: \"kubernetes.io/projected/a94dd682-8a3b-4067-9414-3b6041b34051-kube-api-access-6cxgt\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:42.061777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.061735    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:42.069792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.069339    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls\") pod \"maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:42.105436 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.105404    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:15:42.105623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.105604    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 29 23:15:42.105679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.105659    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 29 23:15:42.314376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.314293    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:15:42.315027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.314991    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:15:42.337807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.336967    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:42.874806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:42.873541    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"]
Oct 29 23:15:43.100599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.100460    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:15:43.100599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.100577    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:15:43.105598 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.105572    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 29 23:15:43.105728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.105625    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 29 23:15:43.254620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.254590    4827 generic.go:356] "Generic (PLEG): container finished" podID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerID="e97f672fd604ff3bfb1e0ad501d13fe417e11679a6c8e4e3329480561ace7b08" exitCode=0
Oct 29 23:15:43.254921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.254893    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" event={"ID":"633e2112-6693-4fdb-980a-8b47b2cd7898","Type":"ContainerDied","Data":"e97f672fd604ff3bfb1e0ad501d13fe417e11679a6c8e4e3329480561ace7b08"}
Oct 29 23:15:43.256746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.256718    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd" event={"ID":"a94dd682-8a3b-4067-9414-3b6041b34051","Type":"ContainerStarted","Data":"56faad88deb2ded253fb39e9ee76abc1c859ea10f4293a12efc5ced33cc26098"}
Oct 29 23:15:43.498660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.498632    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.617433    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-credential-socket\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.617532    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-socket\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.617614    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-podinfo\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.617812    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-token\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-token\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.617917    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-kube-api-access-hz98k\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-kube-api-access-hz98k\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.617968    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-data\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.618037    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-certs\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.618138    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-envoy\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.618257    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633e2112-6693-4fdb-980a-8b47b2cd7898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633e2112-6693-4fdb-980a-8b47b2cd7898-istiod-ca-cert\") pod \"633e2112-6693-4fdb-980a-8b47b2cd7898\" (UID: \"633e2112-6693-4fdb-980a-8b47b2cd7898\") "
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.619599    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633e2112-6693-4fdb-980a-8b47b2cd7898-istiod-ca-cert" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:43.620788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.619934    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-credential-socket" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:43.639968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.639125    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-data" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:43.641045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.641016    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-certs" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:43.642058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.642026    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-socket" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:43.651797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.650627    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-token" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:43.651797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.651107    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-kube-api-access-hz98k" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "kube-api-access-hz98k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:43.700973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.700907    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-podinfo" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:43.727737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727697    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.727737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727738    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz98k\" (UniqueName: \"kubernetes.io/projected/633e2112-6693-4fdb-980a-8b47b2cd7898-kube-api-access-hz98k\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.728016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727753    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.728016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727789    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.728016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727803    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633e2112-6693-4fdb-980a-8b47b2cd7898-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.728016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727817    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.728016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727832    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.728016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.727847    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:43.730695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.730224    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-envoy" pod "633e2112-6693-4fdb-980a-8b47b2cd7898" (UID: "633e2112-6693-4fdb-980a-8b47b2cd7898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:43.829180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:43.829143    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633e2112-6693-4fdb-980a-8b47b2cd7898-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:15:44.106223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.106142    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 29 23:15:44.106223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.106205    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 29 23:15:44.291349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.291309    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd" event={"ID":"a94dd682-8a3b-4067-9414-3b6041b34051","Type":"ContainerStarted","Data":"027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8"}
Oct 29 23:15:44.291564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.291402    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:44.293506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.293476    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"
Oct 29 23:15:44.293637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.293512    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4" event={"ID":"633e2112-6693-4fdb-980a-8b47b2cd7898","Type":"ContainerDied","Data":"5745e8feb04a1607d98b4badbec4aed6d5ca115ba8e588a51a602d3089b34290"}
Oct 29 23:15:44.293637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.293566    4827 scope.go:122] "RemoveContainer" containerID="e97f672fd604ff3bfb1e0ad501d13fe417e11679a6c8e4e3329480561ace7b08"
Oct 29 23:15:44.338078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.338031    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd" podStartSLOduration=2.923521857 podStartE2EDuration="3.338018761s" totalImagesPullingTime="414.496904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:44.333088867 +0000 UTC m=+1979.955999888" watchObservedRunningTime="2026-10-29 23:15:44.338018761 +0000 UTC m=+1979.960929826"
Oct 29 23:15:44.391509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.391471    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"]
Oct 29 23:15:44.395957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.395931    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g6bj4"]
Oct 29 23:15:44.605837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:44.605804    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" path="/var/lib/kubelet/pods/633e2112-6693-4fdb-980a-8b47b2cd7898/volumes"
Oct 29 23:15:45.106589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:45.106552    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 29 23:15:45.106842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:45.106608    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 29 23:15:45.721665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:45.648537    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"]
Oct 29 23:15:46.106347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:46.106269    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 29 23:15:46.106347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:46.106323    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 29 23:15:47.113693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:47.113658    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 29 23:15:47.114722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:47.114315    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 29 23:15:48.302852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:48.302815    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:15:48.303578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:48.303549    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:15:48.306151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:48.306126    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:15:49.952709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:49.952674    4827 scope.go:122] "RemoveContainer" containerID="ce0b7b972dfccc0c6dc8ee518bb62900e47f7c0c0f2c3a0c7ae79d570d38dca4"
Oct 29 23:15:49.964579 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:49.964554    4827 scope.go:122] "RemoveContainer" containerID="e9fe8b31b25512b326c6ac8acb1bf801df41e039bae3426e8bdfb52fc14f070e"
Oct 29 23:15:49.976206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:49.976189    4827 scope.go:122] "RemoveContainer" containerID="f8a3b44a761fdef5e36db6e2a6c574bef3531ea2630d893656a147c94fcc6fb0"
Oct 29 23:15:49.988409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:49.988384    4827 scope.go:122] "RemoveContainer" containerID="93c8a0117d0a7d7e3cb9842256ab786e450e952f85fbae942cd462f2f70500fb"
Oct 29 23:15:50.310737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:50.310651    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:15:51.208104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.208075    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"]
Oct 29 23:15:51.208865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.208849    4827 cpu_manager.go:414] "containerMap: removing container" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy"
Oct 29 23:15:51.208865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.208866    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy"
Oct 29 23:15:51.209018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.209008    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="633e2112-6693-4fdb-980a-8b47b2cd7898" containerName="istio-proxy"
Oct 29 23:15:51.216867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.213606    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.241114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.241080    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"]
Oct 29 23:15:51.242718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.242684    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9e7cc9-openshift-default-dockercfg-mdms6\""
Oct 29 23:15:51.354851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.354781    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfp7\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-kube-api-access-kkfp7\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.354921    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-token\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.355021    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-certs\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.355057    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istiod-ca-cert\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.355138    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-credential-socket\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.355157    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-podinfo\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.355281    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-data\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.355309    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-socket\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.355373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.355362    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-envoy\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475081    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-data\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475140    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-socket\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475224    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-envoy\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475383    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfp7\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-kube-api-access-kkfp7\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475430    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-token\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475513    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-certs\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475546    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istiod-ca-cert\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475670    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-credential-socket\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.478845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.475703    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-podinfo\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.508705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.490033    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-certs\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.508705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.490610    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istiod-ca-cert\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.508705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.491022    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-data\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.508705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.491253    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-socket\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.509397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.509332    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-credential-socket\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.562838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.552369    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-envoy\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.610061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.574327    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-podinfo\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.611633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.610793    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-token\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.612569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.612519    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfp7\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-kube-api-access-kkfp7\") pod \"e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") " pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:51.840904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:51.840151    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:52.331221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.331185    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"]
Oct 29 23:15:52.331221 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:52.331192    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9723d28e_b337_4131_a6bd_8b7a6da3ff16.slice/crio-a553c92c386b497a0129800e14c0e152962dc2aab469d97728d666b16bb20c38 WatchSource:0}: Error finding container a553c92c386b497a0129800e14c0e152962dc2aab469d97728d666b16bb20c38: Status 404 returned error can't find the container with id a553c92c386b497a0129800e14c0e152962dc2aab469d97728d666b16bb20c38
Oct 29 23:15:52.334393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.334357    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:52.334504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.334420    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:52.334504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.334451    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:15:52.523405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.523356    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" event={"ID":"9723d28e-b337-4131-a6bd-8b7a6da3ff16","Type":"ContainerStarted","Data":"da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678"}
Oct 29 23:15:52.523632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.523429    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" event={"ID":"9723d28e-b337-4131-a6bd-8b7a6da3ff16","Type":"ContainerStarted","Data":"a553c92c386b497a0129800e14c0e152962dc2aab469d97728d666b16bb20c38"}
Oct 29 23:15:52.573828 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.573749    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podStartSLOduration=1.573731465 podStartE2EDuration="1.573731465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:52.567703197 +0000 UTC m=+1988.190614218" watchObservedRunningTime="2026-10-29 23:15:52.573731465 +0000 UTC m=+1988.196642486"
Oct 29 23:15:52.841933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:52.841890    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:53.849738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:53.849709    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:54.539715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.539682    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:54.540693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.540677    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:15:54.551693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.551669    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"]
Oct 29 23:15:54.555707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.555688    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:54.574104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.574081    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e170ed\""
Oct 29 23:15:54.574233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.574219    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e170ed-dockercfg-qsbpm\""
Oct 29 23:15:54.579965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.579939    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"]
Oct 29 23:15:54.618818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.617749    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7j4x\" (UniqueName: \"kubernetes.io/projected/5b0efb42-deeb-4e75-b829-c9fd12d39024-kube-api-access-x7j4x\") pod \"payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:54.618818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.617929    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0efb42-deeb-4e75-b829-c9fd12d39024-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:54.729052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.723071    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7j4x\" (UniqueName: \"kubernetes.io/projected/5b0efb42-deeb-4e75-b829-c9fd12d39024-kube-api-access-x7j4x\") pod \"payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:54.729052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.723165    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0efb42-deeb-4e75-b829-c9fd12d39024-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:54.729052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.724740    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0efb42-deeb-4e75-b829-c9fd12d39024-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:54.969817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:54.947956    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7j4x\" (UniqueName: \"kubernetes.io/projected/5b0efb42-deeb-4e75-b829-c9fd12d39024-kube-api-access-x7j4x\") pod \"payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:55.186345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:55.186270    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:55.523231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:55.523079    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"]
Oct 29 23:15:55.524217 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:55.524190    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0efb42_deeb_4e75_b829_c9fd12d39024.slice/crio-44575a7690ab6fa78ed979d81d82516d35d4232303fc00583714c28c880e19d3 WatchSource:0}: Error finding container 44575a7690ab6fa78ed979d81d82516d35d4232303fc00583714c28c880e19d3: Status 404 returned error can't find the container with id 44575a7690ab6fa78ed979d81d82516d35d4232303fc00583714c28c880e19d3
Oct 29 23:15:55.548367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:55.548336    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" event={"ID":"5b0efb42-deeb-4e75-b829-c9fd12d39024","Type":"ContainerStarted","Data":"44575a7690ab6fa78ed979d81d82516d35d4232303fc00583714c28c880e19d3"}
Oct 29 23:15:56.222452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.222401    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"]
Oct 29 23:15:56.227428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.227404    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.248745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.248707    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"]
Oct 29 23:15:56.253731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.253704    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fba83e\"/\"openshift-service-ca.crt\""
Oct 29 23:15:56.253896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.253749    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fba83e\"/\"kube-root-ca.crt\""
Oct 29 23:15:56.256432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.256404    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fba83e\"/\"default-dockercfg-f9t67\""
Oct 29 23:15:56.256604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.256437    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fba83e\"/\"test-model-fba83e-kserve-self-signed-certs\""
Oct 29 23:15:56.339430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.339396    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-model-cache\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.339613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.339454    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tls-certs\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.339613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.339526    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmkpk\" (UniqueName: \"kubernetes.io/projected/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-kube-api-access-rmkpk\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.339812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.339793    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tmp-dir\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.339945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.339928    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-home\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.340502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.339983    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-dshm\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.441860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.441832    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-dshm\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.441942    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-model-cache\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.441987    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tls-certs\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.442010    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmkpk\" (UniqueName: \"kubernetes.io/projected/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-kube-api-access-rmkpk\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.442208    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tmp-dir\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.442359    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-model-cache\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.442361    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-home\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.442571    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tmp-dir\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.442690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.442617    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-home\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.444484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.444463    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-dshm\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.444709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.444686    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tls-certs\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.465526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.465501    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmkpk\" (UniqueName: \"kubernetes.io/projected/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-kube-api-access-rmkpk\") pod \"test-model-fba83e-kserve-59cdf46669-klg5h\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") " pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.602293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.602183    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:56.767708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.767606    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:15:56.767914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:56.767718    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" event={"ID":"5b0efb42-deeb-4e75-b829-c9fd12d39024","Type":"ContainerStarted","Data":"8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10"}
Oct 29 23:15:57.086350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:57.086291    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" podStartSLOduration=3.086274715 podStartE2EDuration="3.086274715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:56.829552077 +0000 UTC m=+1992.452463098" watchObservedRunningTime="2026-10-29 23:15:57.086274715 +0000 UTC m=+1992.709185744"
Oct 29 23:15:57.091403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:57.091380    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"]
Oct 29 23:15:57.091517 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:57.091411    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c593a0c_7716_4855_8b32_fcdd4e5e3f4f.slice/crio-6427a6cf84810a53e603adf6d18081c21737410bc1b1f1c28384ef7c6cb7e337 WatchSource:0}: Error finding container 6427a6cf84810a53e603adf6d18081c21737410bc1b1f1c28384ef7c6cb7e337: Status 404 returned error can't find the container with id 6427a6cf84810a53e603adf6d18081c21737410bc1b1f1c28384ef7c6cb7e337
Oct 29 23:15:57.751029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:57.750979    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" event={"ID":"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f","Type":"ContainerStarted","Data":"c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f"}
Oct 29 23:15:57.751029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:57.751036    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" event={"ID":"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f","Type":"ContainerStarted","Data":"6427a6cf84810a53e603adf6d18081c21737410bc1b1f1c28384ef7c6cb7e337"}
Oct 29 23:15:57.796673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:57.796625    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" podStartSLOduration=1.7966113419999998 podStartE2EDuration="1.796611342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:57.791932302 +0000 UTC m=+1993.414843321" watchObservedRunningTime="2026-10-29 23:15:57.796611342 +0000 UTC m=+1993.419522362"
Oct 29 23:15:58.329689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.329653    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"]
Oct 29 23:15:58.335930 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.335904    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:58.351653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.351625    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"]
Oct 29 23:15:58.356358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.356334    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9e7cc9-dockercfg-9jhgr\""
Oct 29 23:15:58.375723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.375698    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e7cc9\""
Oct 29 23:15:58.394968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.394940    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:58.395136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.395063    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974bq\" (UniqueName: \"kubernetes.io/projected/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-kube-api-access-974bq\") pod \"payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:58.534850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.526553    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:58.534850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.526689    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974bq\" (UniqueName: \"kubernetes.io/projected/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-kube-api-access-974bq\") pod \"payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:58.552787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.551409    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:58.669525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.669487    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974bq\" (UniqueName: \"kubernetes.io/projected/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-kube-api-access-974bq\") pod \"payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:58.700111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.700070    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:15:58.957345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:58.957269    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:15:59.263384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:59.263353    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"]
Oct 29 23:15:59.263931 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:15:59.263905    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233bbfd9_51f0_4b3e_9451_0f391ed9fb75.slice/crio-3e302aae788e83a6b8ab24330382e9e1ce26ef7be3a16a652bd4790eed13d8ca WatchSource:0}: Error finding container 3e302aae788e83a6b8ab24330382e9e1ce26ef7be3a16a652bd4790eed13d8ca: Status 404 returned error can't find the container with id 3e302aae788e83a6b8ab24330382e9e1ce26ef7be3a16a652bd4790eed13d8ca
Oct 29 23:15:59.816075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:59.816038    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" event={"ID":"233bbfd9-51f0-4b3e-9451-0f391ed9fb75","Type":"ContainerStarted","Data":"7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941"}
Oct 29 23:15:59.816273 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:59.816096    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" event={"ID":"233bbfd9-51f0-4b3e-9451-0f391ed9fb75","Type":"ContainerStarted","Data":"3e302aae788e83a6b8ab24330382e9e1ce26ef7be3a16a652bd4790eed13d8ca"}
Oct 29 23:15:59.859845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:15:59.859758    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" podStartSLOduration=1.859745695 podStartE2EDuration="1.859745695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:59.856042678 +0000 UTC m=+1995.478953698" watchObservedRunningTime="2026-10-29 23:15:59.859745695 +0000 UTC m=+1995.482656715"
Oct 29 23:16:00.823151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:00.823120    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:16:01.007751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.007712    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main" containerID="cri-o://4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c" gracePeriod=45
Oct 29 23:16:01.397343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.397317    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:16:01.540253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540223    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-model-cache\") pod \"318f6e68-89e2-451b-b347-ed085617f894\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") "
Oct 29 23:16:01.540446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540274    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/318f6e68-89e2-451b-b347-ed085617f894-tls-certs\" (UniqueName: \"kubernetes.io/secret/318f6e68-89e2-451b-b347-ed085617f894-tls-certs\") pod \"318f6e68-89e2-451b-b347-ed085617f894\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") "
Oct 29 23:16:01.540446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540306    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318f6e68-89e2-451b-b347-ed085617f894-kube-api-access-rv8p7\" (UniqueName: \"kubernetes.io/projected/318f6e68-89e2-451b-b347-ed085617f894-kube-api-access-rv8p7\") pod \"318f6e68-89e2-451b-b347-ed085617f894\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") "
Oct 29 23:16:01.540446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540360    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-dshm\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-dshm\") pod \"318f6e68-89e2-451b-b347-ed085617f894\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") "
Oct 29 23:16:01.540446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540425    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-tmp-dir\") pod \"318f6e68-89e2-451b-b347-ed085617f894\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") "
Oct 29 23:16:01.540683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540513    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-model-cache" pod "318f6e68-89e2-451b-b347-ed085617f894" (UID: "318f6e68-89e2-451b-b347-ed085617f894"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:01.540727 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540676    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-tmp-dir" pod "318f6e68-89e2-451b-b347-ed085617f894" (UID: "318f6e68-89e2-451b-b347-ed085617f894"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:01.540727 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540694    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-home\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-home\") pod \"318f6e68-89e2-451b-b347-ed085617f894\" (UID: \"318f6e68-89e2-451b-b347-ed085617f894\") "
Oct 29 23:16:01.541005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.540971    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-home" pod "318f6e68-89e2-451b-b347-ed085617f894" (UID: "318f6e68-89e2-451b-b347-ed085617f894"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:01.541460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.541439    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:16:01.541460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.541461    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:16:01.541608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.541469    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:16:01.542805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.542786    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318f6e68-89e2-451b-b347-ed085617f894-kube-api-access-rv8p7" pod "318f6e68-89e2-451b-b347-ed085617f894" (UID: "318f6e68-89e2-451b-b347-ed085617f894"). InnerVolumeSpecName "kube-api-access-rv8p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:01.542905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.542807    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318f6e68-89e2-451b-b347-ed085617f894-tls-certs" pod "318f6e68-89e2-451b-b347-ed085617f894" (UID: "318f6e68-89e2-451b-b347-ed085617f894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:01.542905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.542887    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-dshm" pod "318f6e68-89e2-451b-b347-ed085617f894" (UID: "318f6e68-89e2-451b-b347-ed085617f894"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:01.643454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.643366    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318f6e68-89e2-451b-b347-ed085617f894-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:16:01.643454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.643397    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8p7\" (UniqueName: \"kubernetes.io/projected/318f6e68-89e2-451b-b347-ed085617f894-kube-api-access-rv8p7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:16:01.643454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.643410    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318f6e68-89e2-451b-b347-ed085617f894-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:16:01.831908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.831873    4827 generic.go:356] "Generic (PLEG): container finished" podID="318f6e68-89e2-451b-b347-ed085617f894" containerID="4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c" exitCode=0
Oct 29 23:16:01.832355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.831931    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"
Oct 29 23:16:01.832355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.831952    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" event={"ID":"318f6e68-89e2-451b-b347-ed085617f894","Type":"ContainerDied","Data":"4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c"}
Oct 29 23:16:01.832355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.831991    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd" event={"ID":"318f6e68-89e2-451b-b347-ed085617f894","Type":"ContainerDied","Data":"85a9c8351d8c506ade480f0aad4da94c0d4472b1d6cfd54621d03841cf161adc"}
Oct 29 23:16:01.832355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.832002    4827 scope.go:122] "RemoveContainer" containerID="4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c"
Oct 29 23:16:01.843549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.843520    4827 scope.go:122] "RemoveContainer" containerID="4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c"
Oct 29 23:16:01.843858 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:16:01.843834    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c\": container with ID starting with 4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c not found: ID does not exist" containerID="4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c"
Oct 29 23:16:01.843989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.843862    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c"} err="failed to get container status \"4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c\": rpc error: code = NotFound desc = could not find container \"4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c\": container with ID starting with 4a4eb7a3321e3135d6b607bb4ad3a9942a3a13a452bec55705f4bc60dde85e6c not found: ID does not exist"
Oct 29 23:16:01.886279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.886254    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"]
Oct 29 23:16:01.898405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:01.898350    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-32717a/auth-test-model-32717a-kserve-69bddd4797-6lwsd"]
Oct 29 23:16:02.492591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:02.492561    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318f6e68-89e2-451b-b347-ed085617f894" path="/var/lib/kubelet/pods/318f6e68-89e2-451b-b347-ed085617f894/volumes"
Oct 29 23:16:02.826855 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:16:02.826743    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:16:06.543331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.543300    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"]
Oct 29 23:16:06.544610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.544585    4827 cpu_manager.go:414] "containerMap: removing container" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main"
Oct 29 23:16:06.544610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.544609    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main"
Oct 29 23:16:06.544784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.544755    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="318f6e68-89e2-451b-b347-ed085617f894" containerName="main"
Oct 29 23:16:06.550338 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.550312    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.567060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.567039    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-faa4fc-openshift-default-dockercfg-nq4wh\""
Oct 29 23:16:06.587046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.585032    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"]
Oct 29 23:16:06.620900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.620864    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:16:06.634206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.634181    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:16:06.644377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.644347    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-socket\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.644532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.644511    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-token\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.644696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.644661    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-podinfo\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.644892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.644741    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-credential-socket\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.644892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.644816    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-envoy\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.645027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.644912    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-data\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.645131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.645116    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547cb68c-1fcb-41c4-944f-246ffafd7f21-istiod-ca-cert\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.645197 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.645169    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-certs\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.645312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.645288    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8rd\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-kube-api-access-7k8rd\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.755859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755206    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-token\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.755859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755340    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-podinfo\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.755859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755441    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-credential-socket\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.755859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755527    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-envoy\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.755859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755574    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-data\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.755859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755748    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547cb68c-1fcb-41c4-944f-246ffafd7f21-istiod-ca-cert\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.755859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755819    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-certs\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.756415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.755877    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8rd\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-kube-api-access-7k8rd\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.756415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.756002    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-socket\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.756691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.756649    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-socket\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.769787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.758142    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-credential-socket\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.769787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.758430    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-data\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.769787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.759071    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-certs\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.769787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.764067    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547cb68c-1fcb-41c4-944f-246ffafd7f21-istiod-ca-cert\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.824834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.819681    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-envoy\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.877787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.857378    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-token\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.948871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.866466    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-podinfo\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:06.949112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:06.886536    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8rd\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-kube-api-access-7k8rd\") pod \"e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") " pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:07.170787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:07.160252    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:16:07.174981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:07.173724    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:07.610472 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:16:07.610440    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547cb68c_1fcb_41c4_944f_246ffafd7f21.slice/crio-3fb9aefb0272190a759dbb9fb8462212b2ede3c30b1369ebfb35831dda468d7e WatchSource:0}: Error finding container 3fb9aefb0272190a759dbb9fb8462212b2ede3c30b1369ebfb35831dda468d7e: Status 404 returned error can't find the container with id 3fb9aefb0272190a759dbb9fb8462212b2ede3c30b1369ebfb35831dda468d7e
Oct 29 23:16:07.611290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:07.611266    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"]
Oct 29 23:16:07.613714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:07.613679    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:16:07.613870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:07.613759    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:16:07.613870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:07.613828    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:16:07.753869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:07.753837    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:16:08.052860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:08.052760    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" event={"ID":"547cb68c-1fcb-41c4-944f-246ffafd7f21","Type":"ContainerStarted","Data":"063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f"}
Oct 29 23:16:08.053043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:08.052887    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" event={"ID":"547cb68c-1fcb-41c4-944f-246ffafd7f21","Type":"ContainerStarted","Data":"3fb9aefb0272190a759dbb9fb8462212b2ede3c30b1369ebfb35831dda468d7e"}
Oct 29 23:16:08.102057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:08.101989    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podStartSLOduration=2.101970004 podStartE2EDuration="2.101970004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:08.100141842 +0000 UTC m=+2003.723052861" watchObservedRunningTime="2026-10-29 23:16:08.101970004 +0000 UTC m=+2003.724881025"
Oct 29 23:16:08.174583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:08.174555    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:08.181625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:08.181596    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:09.061470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:09.061441    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:09.062906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:09.062884    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:16:11.858005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:11.857969    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:16:13.800783 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.800732    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"]
Oct 29 23:16:13.878652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.878621    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:13.896278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.896248    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1056cc-serving-cert\""
Oct 29 23:16:13.904600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.904416    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"]
Oct 29 23:16:13.945376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.945342    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-maas-api-tls\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:13.945619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.945591    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-metrics-certs\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:13.945724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.945705    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf5d\" (UniqueName: \"kubernetes.io/projected/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-kube-api-access-mjf5d\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:13.945849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:13.945822    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.058790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.058656    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-metrics-certs\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.065054 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.065022    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf5d\" (UniqueName: \"kubernetes.io/projected/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-kube-api-access-mjf5d\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.065350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.065317    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.065734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.065698    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-maas-api-tls\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.067560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.067534    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.083835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.082898    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-maas-api-tls\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.097734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.093610    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-metrics-certs\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.137533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.137478    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf5d\" (UniqueName: \"kubernetes.io/projected/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-kube-api-access-mjf5d\") pod \"maas-api-e2e-inf-a-1056cc-64875d5f88-njqht\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.216154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.216122    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:14.350182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.350108    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"]
Oct 29 23:16:14.356775 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.356745    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.373877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.373846    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49d87f\"/\"test-model-49d87f-kserve-self-signed-certs\""
Oct 29 23:16:14.374060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.373910    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49d87f\"/\"default-dockercfg-lrjx7\""
Oct 29 23:16:14.374342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.374319    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49d87f\"/\"openshift-service-ca.crt\""
Oct 29 23:16:14.374451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.374363    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49d87f\"/\"kube-root-ca.crt\""
Oct 29 23:16:14.375457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.375434    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"]
Oct 29 23:16:14.474414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.474382    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rww79\" (UniqueName: \"kubernetes.io/projected/2bbc573a-282e-4a93-9c79-9599657f6ae6-kube-api-access-rww79\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.474567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.474426    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-model-cache\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.474705 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.474675    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-home\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.474893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.474823    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-dshm\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.475281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.474989    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc573a-282e-4a93-9c79-9599657f6ae6-tls-certs\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.475281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.475111    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-tmp-dir\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.533545 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:16:14.533517    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7954a9_4f8c_48ee_8c4e_ef3ceed3497e.slice/crio-327d1cb522640e26388a15187a1d11c33b84de8d5b1d7732e5181ec24b7c5f67 WatchSource:0}: Error finding container 327d1cb522640e26388a15187a1d11c33b84de8d5b1d7732e5181ec24b7c5f67: Status 404 returned error can't find the container with id 327d1cb522640e26388a15187a1d11c33b84de8d5b1d7732e5181ec24b7c5f67
Oct 29 23:16:14.533915 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.533894    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"]
Oct 29 23:16:14.577173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.577147    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-dshm\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.577302 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.577260    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc573a-282e-4a93-9c79-9599657f6ae6-tls-certs\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.577410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.577393    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-tmp-dir\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.577656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.577623    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rww79\" (UniqueName: \"kubernetes.io/projected/2bbc573a-282e-4a93-9c79-9599657f6ae6-kube-api-access-rww79\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.577822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.577670    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-model-cache\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.577822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.577713    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-tmp-dir\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.577971 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.577889    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-home\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.578030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.578001    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-model-cache\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.578254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.578234    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-home\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.579571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.579552    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-dshm\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.580073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.580047    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc573a-282e-4a93-9c79-9599657f6ae6-tls-certs\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.601495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.601444    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rww79\" (UniqueName: \"kubernetes.io/projected/2bbc573a-282e-4a93-9c79-9599657f6ae6-kube-api-access-rww79\") pod \"test-model-49d87f-kserve-78745dfbf5-tsz5l\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") " pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:14.707790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:14.707284    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:15.289448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:15.289419    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht" event={"ID":"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e","Type":"ContainerStarted","Data":"327d1cb522640e26388a15187a1d11c33b84de8d5b1d7732e5181ec24b7c5f67"}
Oct 29 23:16:15.466745 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:16:15.466705    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bbc573a_282e_4a93_9c79_9599657f6ae6.slice/crio-e06a17f5c71629ef3a03a2fd558f7c8913f44db2d2205cbb860aafb9ba1da87e WatchSource:0}: Error finding container e06a17f5c71629ef3a03a2fd558f7c8913f44db2d2205cbb860aafb9ba1da87e: Status 404 returned error can't find the container with id e06a17f5c71629ef3a03a2fd558f7c8913f44db2d2205cbb860aafb9ba1da87e
Oct 29 23:16:15.468211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:15.468024    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"]
Oct 29 23:16:16.304576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:16.304539    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" event={"ID":"2bbc573a-282e-4a93-9c79-9599657f6ae6","Type":"ContainerStarted","Data":"2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762"}
Oct 29 23:16:16.305093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:16.304593    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" event={"ID":"2bbc573a-282e-4a93-9c79-9599657f6ae6","Type":"ContainerStarted","Data":"e06a17f5c71629ef3a03a2fd558f7c8913f44db2d2205cbb860aafb9ba1da87e"}
Oct 29 23:16:16.306292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:16.306269    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht" event={"ID":"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e","Type":"ContainerStarted","Data":"8358e4c89339f098434be212660b2a98c637c730c4ccf058b481d24ff3b30a43"}
Oct 29 23:16:16.306392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:16.306329    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:16.351867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:16.351809    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" podStartSLOduration=2.351796683 podStartE2EDuration="2.351796683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:16.346927394 +0000 UTC m=+2011.969838404" watchObservedRunningTime="2026-10-29 23:16:16.351796683 +0000 UTC m=+2011.974707703"
Oct 29 23:16:16.393149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:16.393095    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht" podStartSLOduration=2.947353966 podStartE2EDuration="3.393080219s" totalImagesPullingTime="445.726253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:16.388958403 +0000 UTC m=+2012.011869443" watchObservedRunningTime="2026-10-29 23:16:16.393080219 +0000 UTC m=+2012.015991238"
Oct 29 23:16:16.709176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:16.709146    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:22.321487 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:22.321455    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:16:24.714801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:24.712507    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:25.440788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.420909    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:16:25.440788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.422617    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:16:25.472418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.471853    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:25.576537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.576498    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:25.711903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.690938    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:25.712810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.712200    4827 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" podUID="2bbc573a-282e-4a93-9c79-9599657f6ae6" containerName="main" probeResult="failure" output="Get \"https://10.134.1.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:25.785841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.770802    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:25.833795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.828890    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:25.891790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:25.864529    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:26.016887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.012151    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:16:26.076789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.068092    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" podUID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" containerName="main" probeResult="failure" output="Get \"https://10.134.1.23:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:26.446790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.437905    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:16:26.446790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.438499    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:16:26.473793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.459892    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:16:26.598793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.580834    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:26.803866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.796955    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:26.842279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.806233    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:26.843030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.842993    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:16:26.910361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:26.909177    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:27.003862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.003831    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:16:27.007736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.007708    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:16:27.008003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.007986    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:16:27.008174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.008162    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:16:27.008328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.008315    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:16:27.014617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.014584    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:16:27.015391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.015267    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:16:27.042664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.042231    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:16:27.101848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.100522    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:16:27.110322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.110292    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:16:27.218027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.216224    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:16:27.325788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.325232    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:16:27.353477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.353402    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:16:27.402791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:27.393285    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:16:30.134878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:30.134847    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"]
Oct 29 23:16:30.145293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:30.141690    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8be0dd7-8v4c9"]
Oct 29 23:16:30.558834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:30.558740    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b49b552-73e1-413d-a84c-b234f96289b9" path="/var/lib/kubelet/pods/8b49b552-73e1-413d-a84c-b234f96289b9/volumes"
Oct 29 23:16:32.828268 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:16:32.828229    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:16:34.721015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:34.720984    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:34.732415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:34.732391    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:16:38.454856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:38.454815    4827 generic.go:356] "Generic (PLEG): container finished" podID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerID="27e67eb0e647f75e3d148330e20a1bb6e3253dccb8ce6d11ace6e36b53f0eec5" exitCode=0
Oct 29 23:16:38.455320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:38.454906    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" event={"ID":"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1","Type":"ContainerDied","Data":"27e67eb0e647f75e3d148330e20a1bb6e3253dccb8ce6d11ace6e36b53f0eec5"}
Oct 29 23:16:38.455476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:38.455459    4827 scope.go:122] "RemoveContainer" containerID="27e67eb0e647f75e3d148330e20a1bb6e3253dccb8ce6d11ace6e36b53f0eec5"
Oct 29 23:16:38.457899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:38.457870    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:16:38.457995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:38.457965    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:16:38.458039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:38.458001    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:16:39.469002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:39.468959    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" event={"ID":"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1","Type":"ContainerStarted","Data":"a9d899a8d728de2b8b0de7d56c28cf14058e19401a786ad163ec8abce6af5a05"}
Oct 29 23:16:40.409210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:40.409180    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:16:40.410248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:40.410218    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:16:40.410384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:40.410266    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 29 23:16:41.030927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:41.030886    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"]
Oct 29 23:16:41.042918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:41.042888    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19115b36-zjxf5"]
Oct 29 23:16:41.409079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:41.409050    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:16:41.409272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:41.409109    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 29 23:16:42.409403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:42.409061    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:16:42.409403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:42.409114    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 29 23:16:42.515132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:42.515027    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b44739f-afba-477f-9533-6f9540de648a" path="/var/lib/kubelet/pods/1b44739f-afba-477f-9533-6f9540de648a/volumes"
Oct 29 23:16:43.333795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:43.330922    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:16:43.409887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:43.409852    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:16:43.410283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:43.409908    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 29 23:16:44.409212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:44.409179    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:16:44.409392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:44.409238    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 29 23:16:45.409290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:45.409260    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:16:45.409833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:45.409308    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 29 23:16:46.414477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:46.414445    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:16:46.415700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:16:46.415675    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:17:02.826025 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:17:02.825979    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:17:05.153740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:05.153414    4827 generic.go:356] "Generic (PLEG): container finished" podID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerID="c694795834d6ca69cb9e0b536662a4ea9eaf4da0c7946af8b3eda05d78163185" exitCode=0
Oct 29 23:17:05.153740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:05.153705    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" event={"ID":"80096ee0-b2d5-4fac-9a73-6a05813c0ff0","Type":"ContainerDied","Data":"c694795834d6ca69cb9e0b536662a4ea9eaf4da0c7946af8b3eda05d78163185"}
Oct 29 23:17:05.185877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:05.185841    4827 scope.go:122] "RemoveContainer" containerID="c694795834d6ca69cb9e0b536662a4ea9eaf4da0c7946af8b3eda05d78163185"
Oct 29 23:17:05.271185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:05.242352    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:05.271185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:05.242472    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:05.271185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:05.242527    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:06.238791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:06.238234    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" event={"ID":"80096ee0-b2d5-4fac-9a73-6a05813c0ff0","Type":"ContainerStarted","Data":"ed8554e9ba21c641e2efeb405eeff5d089ae329cdf2d8ecd4f9b39d26114871e"}
Oct 29 23:17:06.495418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:06.495331    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:17:06.496243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:06.496221    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Oct 29 23:17:06.496313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:06.496267    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Oct 29 23:17:07.495562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:07.495524    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Oct 29 23:17:07.496024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:07.495585    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Oct 29 23:17:08.495919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:08.495883    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Oct 29 23:17:08.496344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:08.495945    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Oct 29 23:17:09.500791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:09.498100    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Oct 29 23:17:09.500791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:09.498157    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Oct 29 23:17:10.495444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:10.495414    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Oct 29 23:17:10.495661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:10.495462    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Oct 29 23:17:11.498788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:11.498734    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Oct 29 23:17:11.511684 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:11.511633    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Oct 29 23:17:11.817865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:11.817748    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:17:12.505961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:12.505929    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Oct 29 23:17:12.507797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:12.506614    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Oct 29 23:17:13.501220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:13.501185    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:17:13.502309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:13.502287    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:17:16.963521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:16.963484    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"]
Oct 29 23:17:16.970136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:16.970112    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:16.984944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:16.984919    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b2b71e-dockercfg-zqndz\""
Oct 29 23:17:16.985905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:16.985879    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b2b71e\""
Oct 29 23:17:16.989457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:16.989430    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"]
Oct 29 23:17:17.021160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.021123    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c661a0c-8d4b-4918-a2e1-944661be7cc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:17.021540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.021510    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsbd\" (UniqueName: \"kubernetes.io/projected/5c661a0c-8d4b-4918-a2e1-944661be7cc5-kube-api-access-cvsbd\") pod \"payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:17.234847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.224378    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsbd\" (UniqueName: \"kubernetes.io/projected/5c661a0c-8d4b-4918-a2e1-944661be7cc5-kube-api-access-cvsbd\") pod \"payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:17.249093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.249048    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c661a0c-8d4b-4918-a2e1-944661be7cc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:17.276868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.276536    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c661a0c-8d4b-4918-a2e1-944661be7cc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:17.455864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.451893    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsbd\" (UniqueName: \"kubernetes.io/projected/5c661a0c-8d4b-4918-a2e1-944661be7cc5-kube-api-access-cvsbd\") pod \"payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:17.593833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.585341    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:17.930727 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:17:17.930689    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c661a0c_8d4b_4918_a2e1_944661be7cc5.slice/crio-148a93e6cc5c187a1fa1a230694d60baa779583e7603eec64796a26ccb611085 WatchSource:0}: Error finding container 148a93e6cc5c187a1fa1a230694d60baa779583e7603eec64796a26ccb611085: Status 404 returned error can't find the container with id 148a93e6cc5c187a1fa1a230694d60baa779583e7603eec64796a26ccb611085
Oct 29 23:17:17.931556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:17.931494    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"]
Oct 29 23:17:18.535001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:18.534965    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t" event={"ID":"5c661a0c-8d4b-4918-a2e1-944661be7cc5","Type":"ContainerStarted","Data":"f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d"}
Oct 29 23:17:18.535409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:18.535028    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t" event={"ID":"5c661a0c-8d4b-4918-a2e1-944661be7cc5","Type":"ContainerStarted","Data":"148a93e6cc5c187a1fa1a230694d60baa779583e7603eec64796a26ccb611085"}
Oct 29 23:17:18.535409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:18.535168    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:18.576919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:18.576876    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t" podStartSLOduration=2.576862882 podStartE2EDuration="2.576862882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:18.574730206 +0000 UTC m=+2074.197641227" watchObservedRunningTime="2026-10-29 23:17:18.576862882 +0000 UTC m=+2074.199773901"
Oct 29 23:17:24.613678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.613563    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"]
Oct 29 23:17:24.620257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.620234    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"]
Oct 29 23:17:24.620398 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.620317    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.635241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.635215    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-49d87f-openshift-default-dockercfg-xk5gw\""
Oct 29 23:17:24.739956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.739916    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-certs\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740142    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhnl\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-kube-api-access-shhnl\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740194    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740284    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-token\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740406    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299221fe-bd31-4270-9adb-ff8eaee70ed6-istiod-ca-cert\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740470    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-credential-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740596    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-podinfo\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740634    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-data\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.740739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.740727    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-envoy\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.841924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.841895    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhnl\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-kube-api-access-shhnl\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.841941    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.841983    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-token\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842063    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299221fe-bd31-4270-9adb-ff8eaee70ed6-istiod-ca-cert\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842131    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-credential-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842219    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-podinfo\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842292    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-data\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842352    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-envoy\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842409    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842593    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-credential-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842416    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-certs\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842739    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-certs\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.842931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842918    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299221fe-bd31-4270-9adb-ff8eaee70ed6-istiod-ca-cert\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.843045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.842812    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-data\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.844939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.844915    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-envoy\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.845039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.844952    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-podinfo\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.865008 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.864949    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-token\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.865115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.865098    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhnl\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-kube-api-access-shhnl\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:24.935844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:24.935815    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:25.262594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.262567    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"]
Oct 29 23:17:25.262940 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:17:25.262911    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299221fe_bd31_4270_9adb_ff8eaee70ed6.slice/crio-297acbd6e5c89d05c1410563eb3179822789f93373ca2da920b99db6372177be WatchSource:0}: Error finding container 297acbd6e5c89d05c1410563eb3179822789f93373ca2da920b99db6372177be: Status 404 returned error can't find the container with id 297acbd6e5c89d05c1410563eb3179822789f93373ca2da920b99db6372177be
Oct 29 23:17:25.265924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.265887    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:25.266056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.265955    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:25.266056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.265985    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:25.626724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.626686    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" event={"ID":"299221fe-bd31-4270-9adb-ff8eaee70ed6","Type":"ContainerStarted","Data":"16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3"}
Oct 29 23:17:25.627203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.626749    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" event={"ID":"299221fe-bd31-4270-9adb-ff8eaee70ed6","Type":"ContainerStarted","Data":"297acbd6e5c89d05c1410563eb3179822789f93373ca2da920b99db6372177be"}
Oct 29 23:17:25.685418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.685364    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podStartSLOduration=1.685347743 podStartE2EDuration="1.685347743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:25.684055462 +0000 UTC m=+2081.306966481" watchObservedRunningTime="2026-10-29 23:17:25.685347743 +0000 UTC m=+2081.308258762"
Oct 29 23:17:25.936710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:25.936633    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:26.938257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:26.938222    4827 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Oct 29 23:17:26.938664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:26.938284    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Oct 29 23:17:27.936726 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:27.936688    4827 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Oct 29 23:17:27.936947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:27.936760    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Oct 29 23:17:28.936962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:28.936930    4827 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Oct 29 23:17:28.937393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:28.936991    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Oct 29 23:17:29.593790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:29.589263    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:17:29.936169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:29.936137    4827 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Oct 29 23:17:29.936367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:29.936204    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Oct 29 23:17:30.937062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:30.937027    4827 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Oct 29 23:17:30.937488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:30.937100    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Oct 29 23:17:31.937127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:31.937091    4827 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Oct 29 23:17:31.937533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:31.937149    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Oct 29 23:17:32.829644 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:17:32.829609    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:17:32.946290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:32.944858    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:32.947895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:32.947843    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:32.948176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:32.948162    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:17:47.227085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:47.227054    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:17:47.330139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:47.330109    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:17:49.062850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.062314    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"]
Oct 29 23:17:49.068717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.068684    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.103408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.103376    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"]
Oct 29 23:17:49.109506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.109306    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a37090-openshift-default-dockercfg-9msz9\""
Oct 29 23:17:49.193104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193068    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-socket\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193232    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-podinfo\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193337    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkwm\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-kube-api-access-fvkwm\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193490    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-certs\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193524    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-credential-socket\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193549    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-data\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193622    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istiod-ca-cert\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193702    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-token\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.193875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.193853    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-envoy\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.301175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.301130    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-envoy\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.301552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.301524    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-socket\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.301831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.301809    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-podinfo\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.305922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.305893    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkwm\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-kube-api-access-fvkwm\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.325839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.306241    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-certs\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.332676 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.332651    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-credential-socket\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.332921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.332900    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-data\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.333109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.333090    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istiod-ca-cert\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.339451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.333252    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-token\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.349462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.332433    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-certs\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.349658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.308637    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-socket\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.370855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.339956    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-credential-socket\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.371079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.340193    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-data\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.371208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.340843    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istiod-ca-cert\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.371329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.358258    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-podinfo\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.378582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.378520    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkwm\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-kube-api-access-fvkwm\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.398653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.397092    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-envoy\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.422400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.421247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-token\") pod \"e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") " pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:49.713842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:49.713164    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:50.570009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:50.569980    4827 scope.go:122] "RemoveContainer" containerID="2e0999d04060088260d053debb069595cc05e2f0d686eae5b9ea741a1bd6b09c"
Oct 29 23:17:50.587586 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:50.587530    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:50.587756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:50.587633    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:50.587756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:50.587675    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:50.604274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:50.604238    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"]
Oct 29 23:17:50.614006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:50.613980    4827 scope.go:122] "RemoveContainer" containerID="026491fcd55ee40dc78c2a8176284053d6e5320786a13193345219bed39e4c63"
Oct 29 23:17:50.626683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:50.626659    4827 scope.go:122] "RemoveContainer" containerID="5fe0a02edc0589df8ccebf426229451932bc43fcf9609d6cde236b79a8e5f595"
Oct 29 23:17:51.550704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:51.550667    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" event={"ID":"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e","Type":"ContainerStarted","Data":"1f409d85d3ffa9b8a286a6ca3c0bfd865fd46f45ae3c91c8f0034a90cd19d2dd"}
Oct 29 23:17:51.550975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:51.550733    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" event={"ID":"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e","Type":"ContainerStarted","Data":"a047dcf4f015d736aebd18f03d856e3be6dd91df102a39775f4bbd30a84bd951"}
Oct 29 23:17:51.729852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:51.715462    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:51.756608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:51.754330    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:51.933110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:51.931723    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podStartSLOduration=2.931702796 podStartE2EDuration="2.931702796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:51.647043428 +0000 UTC m=+2107.269954445" watchObservedRunningTime="2026-10-29 23:17:51.931702796 +0000 UTC m=+2107.554613797"
Oct 29 23:17:52.561738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.561641    4827 generic.go:356] "Generic (PLEG): container finished" podID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerID="da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678" exitCode=0
Oct 29 23:17:52.561738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.561717    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" event={"ID":"9723d28e-b337-4131-a6bd-8b7a6da3ff16","Type":"ContainerDied","Data":"da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678"}
Oct 29 23:17:52.562265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.562241    4827 scope.go:122] "RemoveContainer" containerID="da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678"
Oct 29 23:17:52.565479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.565432    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:52.565636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.565546    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:52.565636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.565583    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:52.567299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.567273    4827 generic.go:356] "Generic (PLEG): container finished" podID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerID="063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f" exitCode=0
Oct 29 23:17:52.567452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.567354    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" event={"ID":"547cb68c-1fcb-41c4-944f-246ffafd7f21","Type":"ContainerDied","Data":"063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f"}
Oct 29 23:17:52.567742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.567710    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:52.567974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.567937    4827 scope.go:122] "RemoveContainer" containerID="063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f"
Oct 29 23:17:52.569178 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.569159    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:17:52.574230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.574193    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:52.574330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.574289    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:52.574387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:52.574335    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:17:53.588649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:53.588604    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" event={"ID":"9723d28e-b337-4131-a6bd-8b7a6da3ff16","Type":"ContainerStarted","Data":"e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060"}
Oct 29 23:17:53.592740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:53.592659    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" event={"ID":"547cb68c-1fcb-41c4-944f-246ffafd7f21","Type":"ContainerStarted","Data":"696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc"}
Oct 29 23:17:53.842146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:53.842071    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:17:53.843066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:53.843042    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:17:53.843162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:53.843085    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:17:54.061562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:54.061528    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:17:54.175198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:54.175164    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:17:54.176744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:54.176672    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:17:54.177004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:54.176853    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:17:54.540168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:54.540090    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:17:54.842662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:54.842584    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:17:54.842662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:54.842637    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:17:55.215788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:55.215051    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:17:55.219878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:55.219836    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:17:55.843488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:55.843452    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:17:55.844181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:55.844151    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:17:56.235788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:56.205950    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:17:56.235788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:56.206016    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:17:56.850436 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:56.850403    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:17:56.851383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:56.851330    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:17:57.012824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:57.012715    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"]
Oct 29 23:17:57.025104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:57.025063    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy" containerID="cri-o://dd1e16b3401b8f6d3fc3f43fd69194ccfa34622beffed7a40593bc82412b1e2f" gracePeriod=30
Oct 29 23:17:57.176872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:57.176834    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:17:57.177174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:57.177143    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:17:57.879791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:57.874968    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:17:57.879791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:57.875033    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:17:58.202850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:58.202370    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:17:58.202850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:58.202446    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:17:58.842793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:58.842730    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:17:58.842974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:58.842822    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:17:59.178351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:59.178315    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:17:59.178926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:59.178894    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:17:59.858794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:59.856986    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:17:59.858794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:17:59.857042    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:18:00.188792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:00.188173    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:18:00.188792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:00.188253    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:18:00.845157 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:00.845120    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 29 23:18:00.845497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:00.845464    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 29 23:18:01.203384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:01.187332    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 29 23:18:01.203384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:01.187440    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 29 23:18:01.848203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:01.848170    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:18:01.849658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:01.849635    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:18:02.181495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.181467    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:18:02.182858 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.182834    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:18:02.241934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.241900    4827 generic.go:356] "Generic (PLEG): container finished" podID="6f528dc4-4901-428a-8f32-591204781c07" containerID="dd1e16b3401b8f6d3fc3f43fd69194ccfa34622beffed7a40593bc82412b1e2f" exitCode=0
Oct 29 23:18:02.242311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.241967    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" event={"ID":"6f528dc4-4901-428a-8f32-591204781c07","Type":"ContainerDied","Data":"dd1e16b3401b8f6d3fc3f43fd69194ccfa34622beffed7a40593bc82412b1e2f"}
Oct 29 23:18:02.414816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.414786    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:18:02.541642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.541603    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f528dc4-4901-428a-8f32-591204781c07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f528dc4-4901-428a-8f32-591204781c07-istiod-ca-cert\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.541869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.541690    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-envoy\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.541869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.541790    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-kube-api-access-72v4w\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-kube-api-access-72v4w\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.541995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.541905    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-istio-token\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-istio-token\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.542061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542018    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f528dc4-4901-428a-8f32-591204781c07-istiod-ca-cert" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:18:02.542200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542177    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-socket\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.542310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542234    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-credential-socket\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.542310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542276    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-certs\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.542706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542483    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-data\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.542706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542646    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-credential-socket" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:02.542899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542739    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-certs" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:02.542899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542864    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f528dc4-4901-428a-8f32-591204781c07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f528dc4-4901-428a-8f32-591204781c07-istio-podinfo\") pod \"6f528dc4-4901-428a-8f32-591204781c07\" (UID: \"6f528dc4-4901-428a-8f32-591204781c07\") "
Oct 29 23:18:02.542899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.542886    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-data" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:02.543337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.543314    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-socket" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:02.543690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.543673    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.543791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.543690    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.543791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.543700    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.543791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.543708    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.543791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.543717    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f528dc4-4901-428a-8f32-591204781c07-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.544362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.544342    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-kube-api-access-72v4w" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "kube-api-access-72v4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:02.544670 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.544648    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-envoy" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:02.545129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.545104    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f528dc4-4901-428a-8f32-591204781c07-istio-podinfo" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:18:02.545362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.545347    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-istio-token" pod "6f528dc4-4901-428a-8f32-591204781c07" (UID: "6f528dc4-4901-428a-8f32-591204781c07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:02.646223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.646193    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f528dc4-4901-428a-8f32-591204781c07-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.646223 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.646216    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72v4w\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-kube-api-access-72v4w\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.646495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.646244    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f528dc4-4901-428a-8f32-591204781c07-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.646495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:02.646267    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f528dc4-4901-428a-8f32-591204781c07-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:18:02.827267 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:18:02.827222    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:18:03.293833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:03.278354    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6" event={"ID":"6f528dc4-4901-428a-8f32-591204781c07","Type":"ContainerDied","Data":"d21c5cfc30a080715ab2e62be95e5a126a15064dab84054a5560d02d658ae5a2"}
Oct 29 23:18:03.293833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:03.278435    4827 scope.go:122] "RemoveContainer" containerID="dd1e16b3401b8f6d3fc3f43fd69194ccfa34622beffed7a40593bc82412b1e2f"
Oct 29 23:18:03.293833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:03.278629    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"
Oct 29 23:18:03.411782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:03.411733    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"]
Oct 29 23:18:03.435120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:03.435094    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dgqx6"]
Oct 29 23:18:04.008284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.008244    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"]
Oct 29 23:18:04.009567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.009541    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy"
Oct 29 23:18:04.009567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.009564    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy"
Oct 29 23:18:04.009845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.009803    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f528dc4-4901-428a-8f32-591204781c07" containerName="istio-proxy"
Oct 29 23:18:04.015911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.015885    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.042134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.040606    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"]
Oct 29 23:18:04.095459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.095428    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1a8107-openshift-default-dockercfg-9g58t\""
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.202988    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-credential-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203097    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b345f7-bf4d-4b94-9da2-236481c674db-istiod-ca-cert\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203169    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203193    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-istio-token\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203271    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8x8r\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-kube-api-access-f8x8r\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203346    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-envoy\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203474    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-data\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203503    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b345f7-bf4d-4b94-9da2-236481c674db-istio-podinfo\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.207789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.203678    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-certs\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.646789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.628205    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b345f7-bf4d-4b94-9da2-236481c674db-istio-podinfo\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.675377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.675339    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-certs\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.675791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.675747    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-credential-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.676282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.676036    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b345f7-bf4d-4b94-9da2-236481c674db-istiod-ca-cert\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.711914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.700119    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.711914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.700164    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-istio-token\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.711914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.700176    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-certs\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.711914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.700314    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x8r\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-kube-api-access-f8x8r\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.711914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.700456    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-envoy\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.711914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.699988    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b345f7-bf4d-4b94-9da2-236481c674db-istiod-ca-cert\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.711914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.700670    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-data\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.747215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.713012    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.747215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.726596    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f528dc4-4901-428a-8f32-591204781c07" path="/var/lib/kubelet/pods/6f528dc4-4901-428a-8f32-591204781c07/volumes"
Oct 29 23:18:04.747215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.743456    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-credential-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.747215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.744494    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-istio-token\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.769693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.769647    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b345f7-bf4d-4b94-9da2-236481c674db-istio-podinfo\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.774040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.772350    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x8r\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-kube-api-access-f8x8r\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.774040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.773077    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-envoy\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.777637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.775623    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-data\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:04.940570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:04.940494    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:05.274395 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:18:05.274361    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b345f7_bf4d_4b94_9da2_236481c674db.slice/crio-2ec8774388bea7d40a3df1386a572105296c0a65d86abec3aeeb7aaf896002c4 WatchSource:0}: Error finding container 2ec8774388bea7d40a3df1386a572105296c0a65d86abec3aeeb7aaf896002c4: Status 404 returned error can't find the container with id 2ec8774388bea7d40a3df1386a572105296c0a65d86abec3aeeb7aaf896002c4
Oct 29 23:18:05.274549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.274438    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"]
Oct 29 23:18:05.278554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.278510    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:18:05.278653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.278605    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:18:05.278694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.278651    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:18:05.727012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.726975    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" event={"ID":"80b345f7-bf4d-4b94-9da2-236481c674db","Type":"ContainerStarted","Data":"a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7"}
Oct 29 23:18:05.727520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.727041    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" event={"ID":"80b345f7-bf4d-4b94-9da2-236481c674db","Type":"ContainerStarted","Data":"2ec8774388bea7d40a3df1386a572105296c0a65d86abec3aeeb7aaf896002c4"}
Oct 29 23:18:05.798955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.798908    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podStartSLOduration=2.798895181 podStartE2EDuration="2.798895181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:05.783367757 +0000 UTC m=+2121.406278777" watchObservedRunningTime="2026-10-29 23:18:05.798895181 +0000 UTC m=+2121.421806201"
Oct 29 23:18:05.942707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:05.942671    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:06.947059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:06.947033    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:07.744154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:07.744121    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:07.744987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:07.744968    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:18:32.826076 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:18:32.826037    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:18:57.920211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:18:57.919538    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"]
Oct 29 23:19:03.134577 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:19:03.134541    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:19:07.650198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.650165    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"]
Oct 29 23:19:07.656568 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.656541    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.676672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.676637    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"]
Oct 29 23:19:07.680957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.680929    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f4ce70\"/\"default-dockercfg-ghcl8\""
Oct 29 23:19:07.681164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.681041    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f4ce70\"/\"test-model-f4ce70-kserve-self-signed-certs\""
Oct 29 23:19:07.684552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.684521    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f4ce70\"/\"kube-root-ca.crt\""
Oct 29 23:19:07.684702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.684677    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f4ce70\"/\"openshift-service-ca.crt\""
Oct 29 23:19:07.754834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.754800    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tmp-dir\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.755028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.755002    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgdjq\" (UniqueName: \"kubernetes.io/projected/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-kube-api-access-xgdjq\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.755184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.755165    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-home\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.755243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.755215    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-model-cache\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.755370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.755349    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-dshm\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.755463 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.755447    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tls-certs\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.857506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.857473    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgdjq\" (UniqueName: \"kubernetes.io/projected/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-kube-api-access-xgdjq\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.857691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.857642    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-home\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.857758 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.857707    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-model-cache\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.857842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.857817    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-dshm\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.857900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.857874    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tls-certs\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.857974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.857960    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tmp-dir\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.858151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.858128    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-home\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.858239 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.858204    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-model-cache\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.858290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.858274    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tmp-dir\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.860229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.860203    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-dshm\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.860390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.860373    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tls-certs\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.882071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.882043    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgdjq\" (UniqueName: \"kubernetes.io/projected/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-kube-api-access-xgdjq\") pod \"test-model-f4ce70-kserve-6b8cbf9f7c-6d76c\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") " pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:07.983918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:07.979452    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:08.499453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:08.499421    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"]
Oct 29 23:19:08.499619 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:19:08.499489    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod962329a0_0ecc_4b05_a4f5_46ae3ec9a2b5.slice/crio-940001d7a8a14bf8496e4240363cd45a4b54ba8f1bc7bb383ce62763887a3b6e WatchSource:0}: Error finding container 940001d7a8a14bf8496e4240363cd45a4b54ba8f1bc7bb383ce62763887a3b6e: Status 404 returned error can't find the container with id 940001d7a8a14bf8496e4240363cd45a4b54ba8f1bc7bb383ce62763887a3b6e
Oct 29 23:19:08.550740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:08.550709    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" event={"ID":"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5","Type":"ContainerStarted","Data":"940001d7a8a14bf8496e4240363cd45a4b54ba8f1bc7bb383ce62763887a3b6e"}
Oct 29 23:19:09.560688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:09.560650    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" event={"ID":"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5","Type":"ContainerStarted","Data":"b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd"}
Oct 29 23:19:09.604033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:09.603979    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" podStartSLOduration=2.603965986 podStartE2EDuration="2.603965986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:09.60302167 +0000 UTC m=+2185.225932691" watchObservedRunningTime="2026-10-29 23:19:09.603965986 +0000 UTC m=+2185.226877006"
Oct 29 23:19:09.980714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:09.980683    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:12.977261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:12.975701    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" podUID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" containerName="main" containerID="cri-o://c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f" gracePeriod=45
Oct 29 23:19:13.056478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.056440    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" podUID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" containerName="main" probeResult="failure" output="Get \"https://10.134.1.23:8000/ready\": dial tcp 10.134.1.23:8000: connect: connection refused"
Oct 29 23:19:13.292162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.292133    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:19:13.348233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.348192    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-kube-api-access-rmkpk\" (UniqueName: \"kubernetes.io/projected/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-kube-api-access-rmkpk\") pod \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") "
Oct 29 23:19:13.348435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.348383    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tls-certs\") pod \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") "
Oct 29 23:19:13.348435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.348426    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-dshm\") pod \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") "
Oct 29 23:19:13.348571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.348470    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-model-cache\") pod \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") "
Oct 29 23:19:13.348643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.348625    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tmp-dir\") pod \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") "
Oct 29 23:19:13.349289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.348713    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-home\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-home\") pod \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\" (UID: \"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f\") "
Oct 29 23:19:13.349289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.349237    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tmp-dir" pod "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" (UID: "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:13.349505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.349362    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-home" pod "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" (UID: "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:13.349505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.349474    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-model-cache" pod "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" (UID: "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:13.349950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.349928    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:13.350066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.349952    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:13.350066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.349965    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:13.351554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.351286    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-kube-api-access-rmkpk" pod "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" (UID: "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f"). InnerVolumeSpecName "kube-api-access-rmkpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:19:13.352517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.352483    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-dshm" pod "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" (UID: "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:13.352642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.352515    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tls-certs" pod "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" (UID: "6c593a0c-7716-4855-8b32-fcdd4e5e3f4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:19:13.452155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.452120    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmkpk\" (UniqueName: \"kubernetes.io/projected/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-kube-api-access-rmkpk\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:13.452155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.452156    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:13.452372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.452172    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:13.629616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.629585    4827 generic.go:356] "Generic (PLEG): container finished" podID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" containerID="c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f" exitCode=0
Oct 29 23:19:13.629838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.629624    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" event={"ID":"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f","Type":"ContainerDied","Data":"c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f"}
Oct 29 23:19:13.629838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.629630    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"
Oct 29 23:19:13.629838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.629651    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h" event={"ID":"6c593a0c-7716-4855-8b32-fcdd4e5e3f4f","Type":"ContainerDied","Data":"6427a6cf84810a53e603adf6d18081c21737410bc1b1f1c28384ef7c6cb7e337"}
Oct 29 23:19:13.629838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.629667    4827 scope.go:122] "RemoveContainer" containerID="c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f"
Oct 29 23:19:13.641707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.641688    4827 scope.go:122] "RemoveContainer" containerID="c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f"
Oct 29 23:19:13.642043 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:19:13.642022    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f\": container with ID starting with c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f not found: ID does not exist" containerID="c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f"
Oct 29 23:19:13.642117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.642050    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f"} err="failed to get container status \"c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f\": rpc error: code = NotFound desc = could not find container \"c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f\": container with ID starting with c40977327cb8cb939feba79c4d1216832bc01965953ca9e176c916a34742655f not found: ID does not exist"
Oct 29 23:19:13.682628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.682601    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"]
Oct 29 23:19:13.694641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:13.694615    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fba83e/test-model-fba83e-kserve-59cdf46669-klg5h"]
Oct 29 23:19:14.573260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:14.570166    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" path="/var/lib/kubelet/pods/6c593a0c-7716-4855-8b32-fcdd4e5e3f4f/volumes"
Oct 29 23:19:17.980231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:17.980200    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:17.992703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:17.992679    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:18.753074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:18.753048    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:19:23.589856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:23.589824    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"]
Oct 29 23:19:31.525752 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.525675    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"]
Oct 29 23:19:31.527146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.527117    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" containerName="main"
Oct 29 23:19:31.527146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.527139    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" containerName="main"
Oct 29 23:19:31.527361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.527275    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c593a0c-7716-4855-8b32-fcdd4e5e3f4f" containerName="main"
Oct 29 23:19:31.534359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.534304    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.555874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.555845    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"]
Oct 29 23:19:31.557160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.557138    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1056cc\"/\"kube-root-ca.crt\""
Oct 29 23:19:31.557387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.557371    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1056cc\"/\"openshift-service-ca.crt\""
Oct 29 23:19:31.563026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.562988    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1056cc\"/\"default-dockercfg-t4lks\""
Oct 29 23:19:31.563026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.563007    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1056cc\"/\"test-model-1056cc-kserve-self-signed-certs\""
Oct 29 23:19:31.649806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.649756    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-model-cache\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.649998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.649838    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-dshm\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.649998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.649978    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c9dd68-a81b-44ed-9430-b0168a557e39-tls-certs\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.650122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.650048    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-tmp-dir\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.650173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.650149    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-home\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.650209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.650171    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djk2s\" (UniqueName: \"kubernetes.io/projected/15c9dd68-a81b-44ed-9430-b0168a557e39-kube-api-access-djk2s\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.751573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.751533    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-home\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.751573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.751574    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djk2s\" (UniqueName: \"kubernetes.io/projected/15c9dd68-a81b-44ed-9430-b0168a557e39-kube-api-access-djk2s\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.751871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.751857    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-model-cache\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.751933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.751892    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-dshm\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.751982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.751959    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c9dd68-a81b-44ed-9430-b0168a557e39-tls-certs\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.752045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.752008    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-tmp-dir\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.752045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.752016    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-home\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.752195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.752173    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-model-cache\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.752351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.752329    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-tmp-dir\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.754204 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.754181    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-dshm\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.754596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.754578    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c9dd68-a81b-44ed-9430-b0168a557e39-tls-certs\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.800186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.800058    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djk2s\" (UniqueName: \"kubernetes.io/projected/15c9dd68-a81b-44ed-9430-b0168a557e39-kube-api-access-djk2s\") pod \"test-model-1056cc-kserve-7f6b498f95-gm4mf\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") " pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:31.863160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:31.863060    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:32.363141 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:32.363111    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"]
Oct 29 23:19:32.363255 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:19:32.363148    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c9dd68_a81b_44ed_9430_b0168a557e39.slice/crio-a9720cc0abe8952a85355af07441bb214156448c4a847496b564275f1457c9c1 WatchSource:0}: Error finding container a9720cc0abe8952a85355af07441bb214156448c4a847496b564275f1457c9c1: Status 404 returned error can't find the container with id a9720cc0abe8952a85355af07441bb214156448c4a847496b564275f1457c9c1
Oct 29 23:19:32.829568 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:19:32.829534    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:19:32.988224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:32.988179    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" event={"ID":"15c9dd68-a81b-44ed-9430-b0168a557e39","Type":"ContainerStarted","Data":"364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe"}
Oct 29 23:19:32.988224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:32.988224    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" event={"ID":"15c9dd68-a81b-44ed-9430-b0168a557e39","Type":"ContainerStarted","Data":"a9720cc0abe8952a85355af07441bb214156448c4a847496b564275f1457c9c1"}
Oct 29 23:19:33.031793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:33.031734    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podStartSLOduration=2.031719583 podStartE2EDuration="2.031719583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:33.027610485 +0000 UTC m=+2208.650521507" watchObservedRunningTime="2026-10-29 23:19:33.031719583 +0000 UTC m=+2208.654630647"
Oct 29 23:19:33.863655 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:33.863621    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:38.087145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:38.087062    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"]
Oct 29 23:19:38.087525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:38.087341    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy" containerID="cri-o://27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd" gracePeriod=30
Oct 29 23:19:38.643960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:38.642542    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" podUID="2bbc573a-282e-4a93-9c79-9599657f6ae6" containerName="main" containerID="cri-o://2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762" gracePeriod=45
Oct 29 23:19:38.723393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:38.723354    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" podUID="2bbc573a-282e-4a93-9c79-9599657f6ae6" containerName="main" probeResult="failure" output="Get \"https://10.134.1.27:8000/ready\": dial tcp 10.134.1.27:8000: connect: connection refused"
Oct 29 23:19:38.986961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:38.986926    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:19:39.049071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.049031    4827 generic.go:356] "Generic (PLEG): container finished" podID="2bbc573a-282e-4a93-9c79-9599657f6ae6" containerID="2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762" exitCode=0
Oct 29 23:19:39.049291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.049100    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"
Oct 29 23:19:39.049291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.049155    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" event={"ID":"2bbc573a-282e-4a93-9c79-9599657f6ae6","Type":"ContainerDied","Data":"2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762"}
Oct 29 23:19:39.049291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.049194    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l" event={"ID":"2bbc573a-282e-4a93-9c79-9599657f6ae6","Type":"ContainerDied","Data":"e06a17f5c71629ef3a03a2fd558f7c8913f44db2d2205cbb860aafb9ba1da87e"}
Oct 29 23:19:39.049291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.049215    4827 scope.go:122] "RemoveContainer" containerID="2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762"
Oct 29 23:19:39.072993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.072970    4827 scope.go:122] "RemoveContainer" containerID="2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762"
Oct 29 23:19:39.073316 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:19:39.073290    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762\": container with ID starting with 2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762 not found: ID does not exist" containerID="2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762"
Oct 29 23:19:39.073434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.073326    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762"} err="failed to get container status \"2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762\": rpc error: code = NotFound desc = could not find container \"2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762\": container with ID starting with 2b9e48eb982a73b7e276077844c087867a2c56a9d1b4e4f9795103b1f6320762 not found: ID does not exist"
Oct 29 23:19:39.140580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.140548    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbc573a-282e-4a93-9c79-9599657f6ae6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc573a-282e-4a93-9c79-9599657f6ae6-tls-certs\") pod \"2bbc573a-282e-4a93-9c79-9599657f6ae6\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") "
Oct 29 23:19:39.141084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.140670    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-tmp-dir\") pod \"2bbc573a-282e-4a93-9c79-9599657f6ae6\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") "
Oct 29 23:19:39.141084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.140806    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-model-cache\") pod \"2bbc573a-282e-4a93-9c79-9599657f6ae6\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") "
Oct 29 23:19:39.141084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.140885    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-home\") pod \"2bbc573a-282e-4a93-9c79-9599657f6ae6\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") "
Oct 29 23:19:39.141084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.140930    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbc573a-282e-4a93-9c79-9599657f6ae6-kube-api-access-rww79\" (UniqueName: \"kubernetes.io/projected/2bbc573a-282e-4a93-9c79-9599657f6ae6-kube-api-access-rww79\") pod \"2bbc573a-282e-4a93-9c79-9599657f6ae6\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") "
Oct 29 23:19:39.141084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.140960    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-dshm\") pod \"2bbc573a-282e-4a93-9c79-9599657f6ae6\" (UID: \"2bbc573a-282e-4a93-9c79-9599657f6ae6\") "
Oct 29 23:19:39.141084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.140995    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-tmp-dir" pod "2bbc573a-282e-4a93-9c79-9599657f6ae6" (UID: "2bbc573a-282e-4a93-9c79-9599657f6ae6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:39.141425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.141100    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-home" pod "2bbc573a-282e-4a93-9c79-9599657f6ae6" (UID: "2bbc573a-282e-4a93-9c79-9599657f6ae6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:39.141425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.141108    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-model-cache" pod "2bbc573a-282e-4a93-9c79-9599657f6ae6" (UID: "2bbc573a-282e-4a93-9c79-9599657f6ae6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:39.141921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.141899    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:39.141921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.141921    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:39.142066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.141934    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:39.143027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.143000    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbc573a-282e-4a93-9c79-9599657f6ae6-tls-certs" pod "2bbc573a-282e-4a93-9c79-9599657f6ae6" (UID: "2bbc573a-282e-4a93-9c79-9599657f6ae6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:19:39.143133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.143071    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbc573a-282e-4a93-9c79-9599657f6ae6-kube-api-access-rww79" pod "2bbc573a-282e-4a93-9c79-9599657f6ae6" (UID: "2bbc573a-282e-4a93-9c79-9599657f6ae6"). InnerVolumeSpecName "kube-api-access-rww79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:19:39.143133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.143072    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-dshm" pod "2bbc573a-282e-4a93-9c79-9599657f6ae6" (UID: "2bbc573a-282e-4a93-9c79-9599657f6ae6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:39.243184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.243110    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rww79\" (UniqueName: \"kubernetes.io/projected/2bbc573a-282e-4a93-9c79-9599657f6ae6-kube-api-access-rww79\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:39.243184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.243139    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc573a-282e-4a93-9c79-9599657f6ae6-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:39.243184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.243154    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc573a-282e-4a93-9c79-9599657f6ae6-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:39.395867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.395755    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"]
Oct 29 23:19:39.406333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:39.406308    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-49d87f/test-model-49d87f-kserve-78745dfbf5-tsz5l"]
Oct 29 23:19:40.491089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:40.491052    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbc573a-282e-4a93-9c79-9599657f6ae6" path="/var/lib/kubelet/pods/2bbc573a-282e-4a93-9c79-9599657f6ae6/volumes"
Oct 29 23:19:41.863865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:41.863829    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:41.877572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:41.877537    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:42.090019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:42.089985    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:19:43.451055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.451029    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:19:43.586407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586370    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-envoy\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.586626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586439    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-socket\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.586626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586481    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-certs\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.586626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586540    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b66c029-ab0c-4a37-82fc-889fb3667b97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b66c029-ab0c-4a37-82fc-889fb3667b97-istiod-ca-cert\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.586626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586619    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-data\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.586977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586726    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-podinfo\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.586977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586790    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-token\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-token\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.587100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.586984    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-kube-api-access-7zhkm\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-kube-api-access-7zhkm\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.587100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.587023    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-credential-socket\") pod \"8b66c029-ab0c-4a37-82fc-889fb3667b97\" (UID: \"8b66c029-ab0c-4a37-82fc-889fb3667b97\") "
Oct 29 23:19:43.587749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.587721    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-data" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:43.587989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.587964    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-credential-socket" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:43.588081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.588028    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-socket" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:43.588081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.588032    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.588345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.588318    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-certs" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:43.588457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.588359    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b66c029-ab0c-4a37-82fc-889fb3667b97-istiod-ca-cert" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:19:43.589753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.589726    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-podinfo" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:19:43.593147 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.593121    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-envoy" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:19:43.593261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.593170    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-kube-api-access-7zhkm" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "kube-api-access-7zhkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:19:43.594523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.594499    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-token" pod "8b66c029-ab0c-4a37-82fc-889fb3667b97" (UID: "8b66c029-ab0c-4a37-82fc-889fb3667b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690065    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b66c029-ab0c-4a37-82fc-889fb3667b97-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690092    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690101    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690110    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhkm\" (UniqueName: \"kubernetes.io/projected/8b66c029-ab0c-4a37-82fc-889fb3667b97-kube-api-access-7zhkm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690119    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690128    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690136    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:43.690148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:43.690144    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b66c029-ab0c-4a37-82fc-889fb3667b97-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:19:44.093737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.093659    4827 generic.go:356] "Generic (PLEG): container finished" podID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerID="27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd" exitCode=0
Oct 29 23:19:44.093737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.093696    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" event={"ID":"8b66c029-ab0c-4a37-82fc-889fb3667b97","Type":"ContainerDied","Data":"27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd"}
Oct 29 23:19:44.093737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.093712    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"
Oct 29 23:19:44.093737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.093737    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m" event={"ID":"8b66c029-ab0c-4a37-82fc-889fb3667b97","Type":"ContainerDied","Data":"7cb7a47e0eb518f1d102bc2499a7c781e813725a89b55914db4accbe491c342f"}
Oct 29 23:19:44.094045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.093758    4827 scope.go:122] "RemoveContainer" containerID="27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd"
Oct 29 23:19:44.106507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.106490    4827 scope.go:122] "RemoveContainer" containerID="2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c"
Oct 29 23:19:44.118062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.118042    4827 scope.go:122] "RemoveContainer" containerID="27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd"
Oct 29 23:19:44.118569 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:19:44.118528    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd\": container with ID starting with 27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd not found: ID does not exist" containerID="27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd"
Oct 29 23:19:44.118707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.118567    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd"} err="failed to get container status \"27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd\": rpc error: code = NotFound desc = could not find container \"27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd\": container with ID starting with 27be52b3555cd34fa475ed7966546fc0c6e13d031370163154d5b7d0c22b9abd not found: ID does not exist"
Oct 29 23:19:44.118707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.118592    4827 scope.go:122] "RemoveContainer" containerID="2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c"
Oct 29 23:19:44.119097 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:19:44.119071    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c\": container with ID starting with 2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c not found: ID does not exist" containerID="2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c"
Oct 29 23:19:44.119226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.119108    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c"} err="failed to get container status \"2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c\": rpc error: code = NotFound desc = could not find container \"2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c\": container with ID starting with 2bef9ba89d5ff06303fc32e22f79a04fe26e6b314bd5d930a398dd49122ff25c not found: ID does not exist"
Oct 29 23:19:44.158116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.158075    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"]
Oct 29 23:19:44.162467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.162440    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-z2b7m"]
Oct 29 23:19:44.492830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:44.492795    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" path="/var/lib/kubelet/pods/8b66c029-ab0c-4a37-82fc-889fb3667b97/volumes"
Oct 29 23:19:57.805386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:57.805345    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:57.848474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:57.847266    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:19:58.028755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.028711    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.049975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.049934    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.093486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.091366    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" probeResult="failure" output="Get \"https://10.134.1.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.123841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.110855    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.126163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.125601    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.226915 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.226869    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 23:19:58.342598 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.342557    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.392794 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.387542    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.398789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.395930    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:19:58.420917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.420871    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.458790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.453844    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 23:19:58.463549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.463511    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.601124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.600861    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.752844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.751953    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.871789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.869891    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:58.871789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.870135    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:19:58.871789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.870668    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:19:58.940831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:58.936358    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 29 23:19:59.120498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.120412    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:19:59.150800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.143839    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.32:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:19:59.268242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.268204    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"]
Oct 29 23:19:59.269727 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.269699    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2bbc573a-282e-4a93-9c79-9599657f6ae6" containerName="main"
Oct 29 23:19:59.269727 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.269725    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbc573a-282e-4a93-9c79-9599657f6ae6" containerName="main"
Oct 29 23:19:59.269995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.269813    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy"
Oct 29 23:19:59.269995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.269821    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy"
Oct 29 23:19:59.269995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.269842    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy"
Oct 29 23:19:59.269995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.269869    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy"
Oct 29 23:19:59.270214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.270128    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy"
Oct 29 23:19:59.270214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.270144    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b66c029-ab0c-4a37-82fc-889fb3667b97" containerName="istio-proxy"
Oct 29 23:19:59.270214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.270161    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbc573a-282e-4a93-9c79-9599657f6ae6" containerName="main"
Oct 29 23:19:59.277129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.277103    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.296946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.296913    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"]
Oct 29 23:19:59.297151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.297077    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-024aa6-openshift-default-dockercfg-hgk8n\""
Oct 29 23:19:59.312857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.312821    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-credential-socket\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313006    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-certs\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313265    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-envoy\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313319    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-data\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313440    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvql\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-kube-api-access-jmvql\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313491    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-socket\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313619    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-istio-token\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313684    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c041e2d-bd10-460e-a154-06946afd4220-istio-podinfo\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.313840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.313782    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c041e2d-bd10-460e-a154-06946afd4220-istiod-ca-cert\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415123    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-envoy\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415174    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-data\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415203    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvql\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-kube-api-access-jmvql\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415238    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-socket\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415303    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-istio-token\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415354    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c041e2d-bd10-460e-a154-06946afd4220-istio-podinfo\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415387    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c041e2d-bd10-460e-a154-06946afd4220-istiod-ca-cert\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415523    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-credential-socket\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415635    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-certs\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415709    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-socket\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.415797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.415635    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-data\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.416047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.416026    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-certs\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.416188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.416165    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c041e2d-bd10-460e-a154-06946afd4220-istiod-ca-cert\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.416306 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.416242    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-credential-socket\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.418576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.418552    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c041e2d-bd10-460e-a154-06946afd4220-istio-podinfo\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.419409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.419386    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-envoy\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.438116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.438083    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-istio-token\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.438265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.438241    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvql\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-kube-api-access-jmvql\") pod \"e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") " pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.601791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.601120    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:19:59.963094 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:19:59.963062    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c041e2d_bd10_460e_a154_06946afd4220.slice/crio-544e70eb1fc6d35007d8f4cc70b6046593b91124ed911e6e6fea97e7a3d82218 WatchSource:0}: Error finding container 544e70eb1fc6d35007d8f4cc70b6046593b91124ed911e6e6fea97e7a3d82218: Status 404 returned error can't find the container with id 544e70eb1fc6d35007d8f4cc70b6046593b91124ed911e6e6fea97e7a3d82218
Oct 29 23:19:59.964550 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.964358    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"]
Oct 29 23:19:59.970708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.970668    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:19:59.970874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.970743    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:19:59.970874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:19:59.970796    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:20:00.409077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:00.409038    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" event={"ID":"6c041e2d-bd10-460e-a154-06946afd4220","Type":"ContainerStarted","Data":"b04db3035bfab780619b1bbacdd4b8fe1ff9b78252b07a8160be3947af93996a"}
Oct 29 23:20:00.409293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:00.409100    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" event={"ID":"6c041e2d-bd10-460e-a154-06946afd4220","Type":"ContainerStarted","Data":"544e70eb1fc6d35007d8f4cc70b6046593b91124ed911e6e6fea97e7a3d82218"}
Oct 29 23:20:00.452853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:00.452801    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podStartSLOduration=1.452785434 podStartE2EDuration="1.452785434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:00.452091566 +0000 UTC m=+2236.075002587" watchObservedRunningTime="2026-10-29 23:20:00.452785434 +0000 UTC m=+2236.075696458"
Oct 29 23:20:00.604887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:00.604852    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:20:01.876791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.869479    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"]
Oct 29 23:20:01.914389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.912705    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:01.914389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.912875    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:20:01.914389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.912941    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"]
Oct 29 23:20:01.928736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.928704    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e7cc9\"/\"default-dockercfg-qnshg\""
Oct 29 23:20:01.929617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.928861    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e7cc9\"/\"kube-root-ca.crt\""
Oct 29 23:20:01.929617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.929051    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e7cc9\"/\"test-model-9e7cc9-kserve-self-signed-certs\""
Oct 29 23:20:01.938998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:01.938972    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e7cc9\"/\"openshift-service-ca.crt\""
Oct 29 23:20:02.006075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.006041    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e0560-7627-4f0d-981e-e607a2d5598d-tls-certs\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.006291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.006094    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-home\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.006291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.006153    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cthhc\" (UniqueName: \"kubernetes.io/projected/fc6e0560-7627-4f0d-981e-e607a2d5598d-kube-api-access-cthhc\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.006291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.006282    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-dshm\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.006451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.006341    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-model-cache\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.006566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.006543    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-tmp-dir\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.108283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.108251    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e0560-7627-4f0d-981e-e607a2d5598d-tls-certs\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.108461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.108307    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-home\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.108461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.108350    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cthhc\" (UniqueName: \"kubernetes.io/projected/fc6e0560-7627-4f0d-981e-e607a2d5598d-kube-api-access-cthhc\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.108461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.108392    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-dshm\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.108461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.108446    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-model-cache\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.108729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.108587    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-tmp-dir\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.108918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.108887    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-model-cache\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.109093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.109046    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-tmp-dir\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.109093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.109080    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-home\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.110905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.110880    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-dshm\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.111139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.111119    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e0560-7627-4f0d-981e-e607a2d5598d-tls-certs\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.131278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.131163    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cthhc\" (UniqueName: \"kubernetes.io/projected/fc6e0560-7627-4f0d-981e-e607a2d5598d-kube-api-access-cthhc\") pod \"test-model-9e7cc9-kserve-69798c4b9f-4vc8x\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") " pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.258791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.253126    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:02.563545 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.562806    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:20:02.563753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.563663    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:20:02.825643 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:20:02.825612    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc6e0560_7627_4f0d_981e_e607a2d5598d.slice/crio-0034748cc25b7b62c6c8d681a8817ef854e41b245d8c3a41e231d7bf728f73d4 WatchSource:0}: Error finding container 0034748cc25b7b62c6c8d681a8817ef854e41b245d8c3a41e231d7bf728f73d4: Status 404 returned error can't find the container with id 0034748cc25b7b62c6c8d681a8817ef854e41b245d8c3a41e231d7bf728f73d4
Oct 29 23:20:02.826734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:02.826704    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"]
Oct 29 23:20:02.828533 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:20:02.828504    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:20:03.359827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:03.359792    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"]
Oct 29 23:20:03.360321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:03.360037    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6" podUID="d9a99063-d0e7-4e39-93c8-b210080df169" containerName="maas-api" containerID="cri-o://3c0010100a7d8cd5b5b46f369f3a7e1f912d4a9980ad238b9cec1451976b24ea" gracePeriod=30
Oct 29 23:20:03.641162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:03.639566    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" event={"ID":"fc6e0560-7627-4f0d-981e-e607a2d5598d","Type":"ContainerStarted","Data":"7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b"}
Oct 29 23:20:03.641162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:03.639625    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" event={"ID":"fc6e0560-7627-4f0d-981e-e607a2d5598d","Type":"ContainerStarted","Data":"0034748cc25b7b62c6c8d681a8817ef854e41b245d8c3a41e231d7bf728f73d4"}
Oct 29 23:20:03.646220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:03.646183    4827 generic.go:356] "Generic (PLEG): container finished" podID="d9a99063-d0e7-4e39-93c8-b210080df169" containerID="3c0010100a7d8cd5b5b46f369f3a7e1f912d4a9980ad238b9cec1451976b24ea" exitCode=0
Oct 29 23:20:03.646892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:03.646864    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6" event={"ID":"d9a99063-d0e7-4e39-93c8-b210080df169","Type":"ContainerDied","Data":"3c0010100a7d8cd5b5b46f369f3a7e1f912d4a9980ad238b9cec1451976b24ea"}
Oct 29 23:20:04.342790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.276524    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:04.614330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.614305    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:20:04.657527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.657467    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podStartSLOduration=3.6574472350000002 podStartE2EDuration="3.657447235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:03.730132306 +0000 UTC m=+2239.353043328" watchObservedRunningTime="2026-10-29 23:20:04.657447235 +0000 UTC m=+2240.280358257"
Oct 29 23:20:04.658819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.658794    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"
Oct 29 23:20:04.658996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.658803    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6" event={"ID":"d9a99063-d0e7-4e39-93c8-b210080df169","Type":"ContainerDied","Data":"61b5285f069685f2701e5e8d3bdf6b6c1076decfb5d5102ee6473b87996fa2e1"}
Oct 29 23:20:04.658996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.658900    4827 scope.go:122] "RemoveContainer" containerID="3c0010100a7d8cd5b5b46f369f3a7e1f912d4a9980ad238b9cec1451976b24ea"
Oct 29 23:20:04.792816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.792757    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a99063-d0e7-4e39-93c8-b210080df169-kube-api-access-l5hlp\" (UniqueName: \"kubernetes.io/projected/d9a99063-d0e7-4e39-93c8-b210080df169-kube-api-access-l5hlp\") pod \"d9a99063-d0e7-4e39-93c8-b210080df169\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") "
Oct 29 23:20:04.792989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.792933    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a99063-d0e7-4e39-93c8-b210080df169-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a99063-d0e7-4e39-93c8-b210080df169-openshift-service-ca\") pod \"d9a99063-d0e7-4e39-93c8-b210080df169\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") "
Oct 29 23:20:04.793057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.793044    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls\") pod \"d9a99063-d0e7-4e39-93c8-b210080df169\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") "
Oct 29 23:20:04.793142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.793127    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-metrics-certs\") pod \"d9a99063-d0e7-4e39-93c8-b210080df169\" (UID: \"d9a99063-d0e7-4e39-93c8-b210080df169\") "
Oct 29 23:20:04.793366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.793332    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a99063-d0e7-4e39-93c8-b210080df169-openshift-service-ca" pod "d9a99063-d0e7-4e39-93c8-b210080df169" (UID: "d9a99063-d0e7-4e39-93c8-b210080df169"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:20:04.794220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.794196    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a99063-d0e7-4e39-93c8-b210080df169-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:04.795236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.795206    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-metrics-certs" pod "d9a99063-d0e7-4e39-93c8-b210080df169" (UID: "d9a99063-d0e7-4e39-93c8-b210080df169"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:20:04.795372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.795322    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls" pod "d9a99063-d0e7-4e39-93c8-b210080df169" (UID: "d9a99063-d0e7-4e39-93c8-b210080df169"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:20:04.795995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.795968    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a99063-d0e7-4e39-93c8-b210080df169-kube-api-access-l5hlp" pod "d9a99063-d0e7-4e39-93c8-b210080df169" (UID: "d9a99063-d0e7-4e39-93c8-b210080df169"). InnerVolumeSpecName "kube-api-access-l5hlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:20:04.895831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.895798    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:04.895831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.895826    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a99063-d0e7-4e39-93c8-b210080df169-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:04.896054 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:04.895839    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5hlp\" (UniqueName: \"kubernetes.io/projected/d9a99063-d0e7-4e39-93c8-b210080df169-kube-api-access-l5hlp\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:05.048089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:05.048051    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"]
Oct 29 23:20:05.056341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:05.056315    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fba83e-dd978fcf7-r6xb6"]
Oct 29 23:20:06.490860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:06.490829    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a99063-d0e7-4e39-93c8-b210080df169" path="/var/lib/kubelet/pods/d9a99063-d0e7-4e39-93c8-b210080df169/volumes"
Oct 29 23:20:08.085793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:08.071116    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"]
Oct 29 23:20:08.148792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:08.129899    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8029f49f-ggmp7"]
Oct 29 23:20:08.495002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:08.494969    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e70f9e-dbc0-4f79-a139-82e07b76d9a6" path="/var/lib/kubelet/pods/65e70f9e-dbc0-4f79-a139-82e07b76d9a6/volumes"
Oct 29 23:20:11.929826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:11.929790    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"]
Oct 29 23:20:11.930450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:11.930402    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy" containerID="cri-o://6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857" gracePeriod=30
Oct 29 23:20:12.263845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:12.260137    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:12.629396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:12.628198    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:12.816266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:12.816230    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:20:13.052588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:13.052511    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"]
Oct 29 23:20:13.058674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:13.058646    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-070c6d14-hgzxb"]
Oct 29 23:20:14.500819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:14.500788    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823210ce-20d4-48dc-99a3-1da8d708c6e7" path="/var/lib/kubelet/pods/823210ce-20d4-48dc-99a3-1da8d708c6e7/volumes"
Oct 29 23:20:17.282903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.282872    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:20:17.298287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298089    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6101a935-9dd1-4075-86ba-087b6c996633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6101a935-9dd1-4075-86ba-087b6c996633-istiod-ca-cert\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.298287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298153    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6101a935-9dd1-4075-86ba-087b6c996633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6101a935-9dd1-4075-86ba-087b6c996633-istio-podinfo\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.298287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298192    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-socket\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298646    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-data\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298758    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-envoy\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298814    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-certs\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298881    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-kube-api-access-mg7kn\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-kube-api-access-mg7kn\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.298984    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-istio-token\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-istio-token\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.299194    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-credential-socket\") pod \"6101a935-9dd1-4075-86ba-087b6c996633\" (UID: \"6101a935-9dd1-4075-86ba-087b6c996633\") "
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.301488    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-data" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.301923    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-certs" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.302323    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6101a935-9dd1-4075-86ba-087b6c996633-istiod-ca-cert" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.308199    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6101a935-9dd1-4075-86ba-087b6c996633-istio-podinfo" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.308409    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-socket" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.308745    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-envoy" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.309073    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-credential-socket" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.311032    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-istio-token" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:20:17.314620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.313895    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-kube-api-access-mg7kn" pod "6101a935-9dd1-4075-86ba-087b6c996633" (UID: "6101a935-9dd1-4075-86ba-087b6c996633"). InnerVolumeSpecName "kube-api-access-mg7kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:20:17.316126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316102    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316126    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6101a935-9dd1-4075-86ba-087b6c996633-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316140    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6101a935-9dd1-4075-86ba-087b6c996633-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316150    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316159    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316171    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316181    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6101a935-9dd1-4075-86ba-087b6c996633-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316189    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7kn\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-kube-api-access-mg7kn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.316269 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.316199    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6101a935-9dd1-4075-86ba-087b6c996633-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:20:17.844060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.844023    4827 generic.go:356] "Generic (PLEG): container finished" podID="6101a935-9dd1-4075-86ba-087b6c996633" containerID="6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857" exitCode=0
Oct 29 23:20:17.844246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.844081    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"
Oct 29 23:20:17.844246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.844098    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" event={"ID":"6101a935-9dd1-4075-86ba-087b6c996633","Type":"ContainerDied","Data":"6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857"}
Oct 29 23:20:17.844246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.844157    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt" event={"ID":"6101a935-9dd1-4075-86ba-087b6c996633","Type":"ContainerDied","Data":"0251643be2fd509925a355112f974333f6a445ae870a8d56173af9522ed4c5b1"}
Oct 29 23:20:17.844246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.844183    4827 scope.go:122] "RemoveContainer" containerID="6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857"
Oct 29 23:20:17.856657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.856636    4827 scope.go:122] "RemoveContainer" containerID="98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f"
Oct 29 23:20:17.866708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.866690    4827 scope.go:122] "RemoveContainer" containerID="6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857"
Oct 29 23:20:17.867002 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:20:17.866979    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857\": container with ID starting with 6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857 not found: ID does not exist" containerID="6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857"
Oct 29 23:20:17.867071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.867010    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857"} err="failed to get container status \"6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857\": rpc error: code = NotFound desc = could not find container \"6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857\": container with ID starting with 6bafa209a4eecaf45443b6f70838e47b696b6f033c4f66707e4d38f0be5e2857 not found: ID does not exist"
Oct 29 23:20:17.867071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.867027    4827 scope.go:122] "RemoveContainer" containerID="98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f"
Oct 29 23:20:17.867268 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:20:17.867250    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f\": container with ID starting with 98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f not found: ID does not exist" containerID="98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f"
Oct 29 23:20:17.867331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.867278    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f"} err="failed to get container status \"98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f\": rpc error: code = NotFound desc = could not find container \"98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f\": container with ID starting with 98a04c4f9549c829cfd2ab37f6a4bf5bbf8112744ff1feae4b4207750391fb3f not found: ID does not exist"
Oct 29 23:20:17.905596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.905567    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"]
Oct 29 23:20:17.915203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:17.915175    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-425jt"]
Oct 29 23:20:18.492812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:18.492749    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6101a935-9dd1-4075-86ba-087b6c996633" path="/var/lib/kubelet/pods/6101a935-9dd1-4075-86ba-087b6c996633/volumes"
Oct 29 23:20:19.868529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:19.868497    4827 generic.go:356] "Generic (PLEG): container finished" podID="80b345f7-bf4d-4b94-9da2-236481c674db" containerID="a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7" exitCode=0
Oct 29 23:20:19.869012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:19.868573    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" event={"ID":"80b345f7-bf4d-4b94-9da2-236481c674db","Type":"ContainerDied","Data":"a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7"}
Oct 29 23:20:19.869095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:19.869080    4827 scope.go:122] "RemoveContainer" containerID="a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7"
Oct 29 23:20:19.871359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:19.871330    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:20:19.871455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:19.871406    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:20:19.871455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:19.871440    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:20:20.898424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:20.898389    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" event={"ID":"80b345f7-bf4d-4b94-9da2-236481c674db","Type":"ContainerStarted","Data":"3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552"}
Oct 29 23:20:20.940742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:20.940717    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:20:20.942261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:20.942233    4827 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:20:20.942430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:20.942295    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Oct 29 23:20:21.966856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:21.965014    4827 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:20:21.966856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:21.965100    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Oct 29 23:20:22.508101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.508060    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"]
Oct 29 23:20:22.508932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.508911    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d9a99063-d0e7-4e39-93c8-b210080df169" containerName="maas-api"
Oct 29 23:20:22.508932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.508930    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a99063-d0e7-4e39-93c8-b210080df169" containerName="maas-api"
Oct 29 23:20:22.509107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.508955    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy"
Oct 29 23:20:22.509107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.508960    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy"
Oct 29 23:20:22.509231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.509130    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy"
Oct 29 23:20:22.509231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.509143    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy"
Oct 29 23:20:22.509231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.509156    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a99063-d0e7-4e39-93c8-b210080df169" containerName="maas-api"
Oct 29 23:20:22.511946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.511928    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.527864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.527842    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f4ce70-openshift-default-dockercfg-qr8z4\""
Oct 29 23:20:22.544110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.544078    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"]
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.673548    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-podinfo\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.673598    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-envoy\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.673661    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.673703    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-token\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.673833    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-credential-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.673949    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxsq\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-kube-api-access-zgxsq\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.673996    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-data\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.674109    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istiod-ca-cert\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.676791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.674153    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-certs\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.744214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.744180    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785056    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxsq\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-kube-api-access-zgxsq\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785133    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-data\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785246    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istiod-ca-cert\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785289    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-certs\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785445    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-podinfo\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785478    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-envoy\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785541    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785581    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-token\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.786013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.785700    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-credential-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.788060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.788030    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-credential-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.788605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.788580    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-certs\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.789104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.789084    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-data\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.789292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.789267    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.789599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.789575    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istiod-ca-cert\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.791411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.791392    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-envoy\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.791673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.791654    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-podinfo\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.811066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.811038    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-token\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.811287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.811268    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxsq\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-kube-api-access-zgxsq\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.826923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.826894    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:22.943166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.941846    4827 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:20:22.943166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:22.941902    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Oct 29 23:20:23.206468 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:20:23.206435    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02be85c9_a940_4c56_a4ce_bcd3fffcc72f.slice/crio-0ce67dd5baa51a87197e126a7a3590dad0692edab42058da1e593a0f21b05a2f WatchSource:0}: Error finding container 0ce67dd5baa51a87197e126a7a3590dad0692edab42058da1e593a0f21b05a2f: Status 404 returned error can't find the container with id 0ce67dd5baa51a87197e126a7a3590dad0692edab42058da1e593a0f21b05a2f
Oct 29 23:20:23.207873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:23.207745    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"]
Oct 29 23:20:23.211445 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:23.211398    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:20:23.211575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:23.211496    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:20:23.211575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:23.211546    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:20:23.952789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:23.950869    4827 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:20:23.952789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:23.950934    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Oct 29 23:20:24.183108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.183067    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" event={"ID":"02be85c9-a940-4c56-a4ce-bcd3fffcc72f","Type":"ContainerStarted","Data":"62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9"}
Oct 29 23:20:24.189214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.189176    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" event={"ID":"02be85c9-a940-4c56-a4ce-bcd3fffcc72f","Type":"ContainerStarted","Data":"0ce67dd5baa51a87197e126a7a3590dad0692edab42058da1e593a0f21b05a2f"}
Oct 29 23:20:24.245839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.235701    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podStartSLOduration=2.235681964 podStartE2EDuration="2.235681964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:24.234203985 +0000 UTC m=+2259.857115005" watchObservedRunningTime="2026-10-29 23:20:24.235681964 +0000 UTC m=+2259.858593190"
Oct 29 23:20:24.827336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.827292    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:24.829516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.829487    4827 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 29 23:20:24.829675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.829572    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 29 23:20:24.941720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.941366    4827 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:20:24.941720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:24.941437    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Oct 29 23:20:25.719864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.719823    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"]
Oct 29 23:20:25.721297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.721274    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy"
Oct 29 23:20:25.721373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.721301    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6101a935-9dd1-4075-86ba-087b6c996633" containerName="istio-proxy"
Oct 29 23:20:25.724653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.724634    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.737391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.737366    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"]
Oct 29 23:20:25.740647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.740624    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftt8\" (UniqueName: \"kubernetes.io/projected/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-kube-api-access-hftt8\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.741018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.740996    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a9ae14\"/\"default-dockercfg-b589j\""
Oct 29 23:20:25.741121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.741090    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a9ae14\"/\"test-model-a9ae14-kserve-self-signed-certs\""
Oct 29 23:20:25.741213 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.741005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tls-certs\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.741469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.741451    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-home\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.741622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.741604    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tmp-dir\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.741795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.741742    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-model-cache\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.741893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.741834    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-dshm\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.752616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.749509    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a9ae14\"/\"kube-root-ca.crt\""
Oct 29 23:20:25.752616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.749596    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a9ae14\"/\"openshift-service-ca.crt\""
Oct 29 23:20:25.827704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.827676    4827 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 29 23:20:25.827974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.827951    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 29 23:20:25.843632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.843595    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tls-certs\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.843814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.843687    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-home\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.843814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.843717    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tmp-dir\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.843814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.843736    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-model-cache\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.843814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.843760    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-dshm\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.844066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.843825    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftt8\" (UniqueName: \"kubernetes.io/projected/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-kube-api-access-hftt8\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.844793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.844742    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-home\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.844910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.844759    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tmp-dir\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.845101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.845079    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-model-cache\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.846494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.846470    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tls-certs\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.847435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.847412    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-dshm\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.918787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.918257    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftt8\" (UniqueName: \"kubernetes.io/projected/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-kube-api-access-hftt8\") pod \"test-model-a9ae14-kserve-7d9bcddbdf-t2g6d\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") " pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:25.985845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.985626    4827 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:20:25.985845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:25.985687    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Oct 29 23:20:26.048792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:26.048077    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:26.723358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:26.722983    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"]
Oct 29 23:20:26.827890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:26.827862    4827 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 29 23:20:26.828044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:26.827918    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 29 23:20:26.941391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:26.941353    4827 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:20:26.941616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:26.941407    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Oct 29 23:20:27.325051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:27.325013    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" event={"ID":"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382","Type":"ContainerStarted","Data":"97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb"}
Oct 29 23:20:27.325237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:27.325066    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" event={"ID":"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382","Type":"ContainerStarted","Data":"423cb02a46cda5c391215b220b8007ec409c1d9691ad2892d0d92639499cadce"}
Oct 29 23:20:27.394934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:27.394869    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" podStartSLOduration=2.3948442070000002 podStartE2EDuration="2.394844207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:27.386180954 +0000 UTC m=+2263.009091981" watchObservedRunningTime="2026-10-29 23:20:27.394844207 +0000 UTC m=+2263.017755254"
Oct 29 23:20:27.828246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:27.828213    4827 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 29 23:20:27.828658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:27.828266    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 29 23:20:27.946365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:27.946334    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:20:27.947441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:27.947414    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:20:28.049277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:28.049246    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:28.827853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:28.827815    4827 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 29 23:20:28.828044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:28.827880    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 29 23:20:29.827890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:29.827862    4827 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 29 23:20:29.828288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:29.827911    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 29 23:20:30.845378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:30.845348    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:30.846695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:30.846665    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:30.847692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:30.847671    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:20:32.868874 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:20:32.868838    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:20:36.049317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:36.049284    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:36.063450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:36.063423    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:20:36.415326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:20:36.415292    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:21:02.509234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.509159    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"]
Oct 29 23:21:02.515481 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.515463    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.536323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.536292    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"]
Oct 29 23:21:02.539479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.539453    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-024aa6-serving-cert\""
Oct 29 23:21:02.569627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.569593    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9ee5-382e-4008-815d-e697f95f8699-openshift-service-ca\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.569840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.569813    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-metrics-certs\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.570019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.569998    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhnq\" (UniqueName: \"kubernetes.io/projected/b9bc9ee5-382e-4008-815d-e697f95f8699-kube-api-access-5qhnq\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.570142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.570119    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-maas-api-tls\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.672153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.672114    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9ee5-382e-4008-815d-e697f95f8699-openshift-service-ca\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.672377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.672219    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-metrics-certs\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.672377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.672291    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhnq\" (UniqueName: \"kubernetes.io/projected/b9bc9ee5-382e-4008-815d-e697f95f8699-kube-api-access-5qhnq\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.672377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.672337    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-maas-api-tls\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.672894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.672869    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9ee5-382e-4008-815d-e697f95f8699-openshift-service-ca\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.674744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.674719    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-maas-api-tls\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.674887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.674856    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-metrics-certs\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.696247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.696224    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhnq\" (UniqueName: \"kubernetes.io/projected/b9bc9ee5-382e-4008-815d-e697f95f8699-kube-api-access-5qhnq\") pod \"maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.870840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:02.863137    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:02.914412 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:21:02.914361    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:21:03.214988 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:21:03.214932    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bc9ee5_382e_4008_815d_e697f95f8699.slice/crio-b11f5f7d332d79186fa0a144627236039cd6dd0233373d18d63169dc73769ca3 WatchSource:0}: Error finding container b11f5f7d332d79186fa0a144627236039cd6dd0233373d18d63169dc73769ca3: Status 404 returned error can't find the container with id b11f5f7d332d79186fa0a144627236039cd6dd0233373d18d63169dc73769ca3
Oct 29 23:21:03.216389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:03.216356    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"]
Oct 29 23:21:03.224951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:03.224829    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:21:03.928609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:03.928579    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp" event={"ID":"b9bc9ee5-382e-4008-815d-e697f95f8699","Type":"ContainerStarted","Data":"b11f5f7d332d79186fa0a144627236039cd6dd0233373d18d63169dc73769ca3"}
Oct 29 23:21:04.938130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:04.938094    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp" event={"ID":"b9bc9ee5-382e-4008-815d-e697f95f8699","Type":"ContainerStarted","Data":"f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e"}
Oct 29 23:21:04.938633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:04.938189    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:04.979615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:04.979552    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp" podStartSLOduration=2.550413886 podStartE2EDuration="2.979534215s" totalImagesPullingTime="429.120329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:04.977387651 +0000 UTC m=+2300.600298673" watchObservedRunningTime="2026-10-29 23:21:04.979534215 +0000 UTC m=+2300.602445239"
Oct 29 23:21:10.972626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:10.972593    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:21:15.829520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:15.829484    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"]
Oct 29 23:21:15.830015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:15.829758    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp" podUID="9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" containerName="maas-api" containerID="cri-o://8b8c7d248d713dde69b0a6bc332055f408b7a82321d0290944f56f623e470b2f" gracePeriod=30
Oct 29 23:21:16.560113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.560082    4827 generic.go:356] "Generic (PLEG): container finished" podID="9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" containerID="8b8c7d248d713dde69b0a6bc332055f408b7a82321d0290944f56f623e470b2f" exitCode=0
Oct 29 23:21:16.688127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.688095    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:21:16.723851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.720576    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp" event={"ID":"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe","Type":"ContainerDied","Data":"8b8c7d248d713dde69b0a6bc332055f408b7a82321d0290944f56f623e470b2f"}
Oct 29 23:21:16.723851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.720656    4827 scope.go:122] "RemoveContainer" containerID="8b8c7d248d713dde69b0a6bc332055f408b7a82321d0290944f56f623e470b2f"
Oct 29 23:21:16.847662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.847631    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-openshift-service-ca\") pod \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") "
Oct 29 23:21:16.856632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.856603    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-maas-api-tls\") pod \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") "
Oct 29 23:21:16.857126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.857106    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-kube-api-access-dkl9v\" (UniqueName: \"kubernetes.io/projected/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-kube-api-access-dkl9v\") pod \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") "
Oct 29 23:21:16.857290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.857277    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-metrics-certs\") pod \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\" (UID: \"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe\") "
Oct 29 23:21:16.858872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.858815    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-openshift-service-ca" pod "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" (UID: "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:16.861730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.861698    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-maas-api-tls" pod "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" (UID: "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:21:16.862669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.862636    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-kube-api-access-dkl9v" pod "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" (UID: "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe"). InnerVolumeSpecName "kube-api-access-dkl9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:16.863325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.863305    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-metrics-certs" pod "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" (UID: "9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:21:16.979318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.972816    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.979318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.972854    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkl9v\" (UniqueName: \"kubernetes.io/projected/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-kube-api-access-dkl9v\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.979318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.972870    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.979318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:16.972883    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:17.570675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:17.570647    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"
Oct 29 23:21:17.570976 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:17.570684    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp" event={"ID":"9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe","Type":"ContainerDied","Data":"8353bba59264a6f955d3203e9d16d2aee63fe001b77fea699b62b0bb81d78a9b"}
Oct 29 23:21:17.621934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:17.621903    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"]
Oct 29 23:21:17.632426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:17.632400    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49d87f-64899dc5cc-wqbvp"]
Oct 29 23:21:18.491528 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:18.491495    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" path="/var/lib/kubelet/pods/9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe/volumes"
Oct 29 23:21:23.094412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:23.094379    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"]
Oct 29 23:21:23.094975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:23.094654    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy" containerID="cri-o://8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01" gracePeriod=30
Oct 29 23:21:24.187147 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:24.187111    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"]
Oct 29 23:21:24.187587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:24.187385    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy" containerID="cri-o://16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3" gracePeriod=30
Oct 29 23:21:28.387227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.387196    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:21:28.486232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.486200    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-credential-socket\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.486511 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.486491    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istiod-ca-cert\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.487955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.487929    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-token\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.488077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.487962    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-credential-socket" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:28.488186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488164    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-podinfo\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.488273 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488206    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istiod-ca-cert" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:28.488336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488315    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-kube-api-access-pv44g\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.488385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488372    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-data\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.488443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488426    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-certs\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.488980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488511    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-envoy\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.488980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488572    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-socket\") pod \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\" (UID: \"3cb1ccfd-7044-4b45-8e8a-4da139673db6\") "
Oct 29 23:21:28.488980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.488707    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-certs" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:28.489615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.489587    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.489717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.489622    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.489717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.489638    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.489912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.489876    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-socket" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:28.490032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.489960    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-data" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:28.490815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.490669    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-token" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:28.490932 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.490830    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-podinfo" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:28.491025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.491008    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-kube-api-access-pv44g" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "kube-api-access-pv44g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:28.491811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.491749    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-envoy" pod "3cb1ccfd-7044-4b45-8e8a-4da139673db6" (UID: "3cb1ccfd-7044-4b45-8e8a-4da139673db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:28.592168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.592134    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.592168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.592158    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-kube-api-access-pv44g\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.592168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.592173    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.592420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.592187    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.592420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.592197    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1ccfd-7044-4b45-8e8a-4da139673db6-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.592420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.592209    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1ccfd-7044-4b45-8e8a-4da139673db6-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:28.716591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.716559    4827 generic.go:356] "Generic (PLEG): container finished" podID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerID="8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01" exitCode=0
Oct 29 23:21:28.716783 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.716608    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"
Oct 29 23:21:28.716783 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.716633    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" event={"ID":"3cb1ccfd-7044-4b45-8e8a-4da139673db6","Type":"ContainerDied","Data":"8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01"}
Oct 29 23:21:28.716783 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.716687    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z" event={"ID":"3cb1ccfd-7044-4b45-8e8a-4da139673db6","Type":"ContainerDied","Data":"66d008a207cde0e65b09199d0927e40d1e92e9e224b82a7d991e51310e26ff78"}
Oct 29 23:21:28.716783 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.716709    4827 scope.go:122] "RemoveContainer" containerID="8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01"
Oct 29 23:21:28.727648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.727628    4827 scope.go:122] "RemoveContainer" containerID="8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01"
Oct 29 23:21:28.728064 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:21:28.728039    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01\": container with ID starting with 8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01 not found: ID does not exist" containerID="8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01"
Oct 29 23:21:28.728165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.728073    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01"} err="failed to get container status \"8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01\": rpc error: code = NotFound desc = could not find container \"8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01\": container with ID starting with 8acf68183ee69870faa87aed0d645aaa421da0be5fea7db659ed2ac0f1d8ca01 not found: ID does not exist"
Oct 29 23:21:28.767211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.767191    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"]
Oct 29 23:21:28.778155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:28.778134    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-rj45z"]
Oct 29 23:21:29.502559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.502531    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:21:29.518280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518251    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-certs\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518323    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-token\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-token\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518346    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-podinfo\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518407    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-kube-api-access-shhnl\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-kube-api-access-shhnl\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518509    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-certs" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:29.518573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518541    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299221fe-bd31-4270-9adb-ff8eaee70ed6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299221fe-bd31-4270-9adb-ff8eaee70ed6-istiod-ca-cert\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518602    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-credential-socket\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518647    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-socket\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518697    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-data\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.518955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518855    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-credential-socket" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:29.518955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518933    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-envoy\") pod \"299221fe-bd31-4270-9adb-ff8eaee70ed6\" (UID: \"299221fe-bd31-4270-9adb-ff8eaee70ed6\") "
Oct 29 23:21:29.519062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.518986    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-socket" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:29.519268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.519245    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299221fe-bd31-4270-9adb-ff8eaee70ed6-istiod-ca-cert" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:29.519514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.519483    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-data" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:29.520222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.520201    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.520345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.520227    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299221fe-bd31-4270-9adb-ff8eaee70ed6-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.520345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.520242    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.520345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.520260    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.520345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.520274    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.520972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.520952    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-token" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:29.521216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.521195    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-kube-api-access-shhnl" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "kube-api-access-shhnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:29.521814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.521778    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-podinfo" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:29.521907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.521891    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-envoy" pod "299221fe-bd31-4270-9adb-ff8eaee70ed6" (UID: "299221fe-bd31-4270-9adb-ff8eaee70ed6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:29.621521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.621489    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.621521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.621519    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.621521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.621529    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhnl\" (UniqueName: \"kubernetes.io/projected/299221fe-bd31-4270-9adb-ff8eaee70ed6-kube-api-access-shhnl\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.621808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.621538    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299221fe-bd31-4270-9adb-ff8eaee70ed6-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:29.727073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.726979    4827 generic.go:356] "Generic (PLEG): container finished" podID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerID="16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3" exitCode=0
Oct 29 23:21:29.727073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.727039    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"
Oct 29 23:21:29.727073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.727060    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" event={"ID":"299221fe-bd31-4270-9adb-ff8eaee70ed6","Type":"ContainerDied","Data":"16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3"}
Oct 29 23:21:29.727408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.727148    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48" event={"ID":"299221fe-bd31-4270-9adb-ff8eaee70ed6","Type":"ContainerDied","Data":"297acbd6e5c89d05c1410563eb3179822789f93373ca2da920b99db6372177be"}
Oct 29 23:21:29.727408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.727193    4827 scope.go:122] "RemoveContainer" containerID="16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3"
Oct 29 23:21:29.739985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.739967    4827 scope.go:122] "RemoveContainer" containerID="16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3"
Oct 29 23:21:29.740281 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:21:29.740263    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3\": container with ID starting with 16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3 not found: ID does not exist" containerID="16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3"
Oct 29 23:21:29.740354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.740295    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3"} err="failed to get container status \"16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3\": rpc error: code = NotFound desc = could not find container \"16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3\": container with ID starting with 16e3386ceca9adb500fe502f18f8cd90d156e71d867bb9cf88aa914c78cc4fb3 not found: ID does not exist"
Oct 29 23:21:29.777325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.777299    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"]
Oct 29 23:21:29.788106 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:29.788084    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-9fz48"]
Oct 29 23:21:30.490868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:30.490838    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" path="/var/lib/kubelet/pods/299221fe-bd31-4270-9adb-ff8eaee70ed6/volumes"
Oct 29 23:21:30.491418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:30.491403    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" path="/var/lib/kubelet/pods/3cb1ccfd-7044-4b45-8e8a-4da139673db6/volumes"
Oct 29 23:21:32.825685 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:21:32.825652    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:21:42.589675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:42.589630    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"]
Oct 29 23:21:42.590335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:42.589909    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy" containerID="cri-o://471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64" gracePeriod=30
Oct 29 23:21:47.890286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.890258    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:21:47.918912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.918642    4827 generic.go:356] "Generic (PLEG): container finished" podID="a8972826-8c7b-466f-9b19-818336fde77c" containerID="471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64" exitCode=0
Oct 29 23:21:47.918912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.918699    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"
Oct 29 23:21:47.918912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.918804    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" event={"ID":"a8972826-8c7b-466f-9b19-818336fde77c","Type":"ContainerDied","Data":"471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64"}
Oct 29 23:21:47.918912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.918854    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf" event={"ID":"a8972826-8c7b-466f-9b19-818336fde77c","Type":"ContainerDied","Data":"0f8b953fc5d5813bd5b8899510cb3cad3ec68e59d89511c2c6704d5a77f68b00"}
Oct 29 23:21:47.918912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.918882    4827 scope.go:122] "RemoveContainer" containerID="471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64"
Oct 29 23:21:47.930155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.930130    4827 scope.go:122] "RemoveContainer" containerID="471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64"
Oct 29 23:21:47.931209 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:21:47.931185    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64\": container with ID starting with 471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64 not found: ID does not exist" containerID="471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64"
Oct 29 23:21:47.931323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:47.931219    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64"} err="failed to get container status \"471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64\": rpc error: code = NotFound desc = could not find container \"471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64\": container with ID starting with 471b686bcc1d9830230db4b79105d0e8b4378c966350671520d6b7291e669a64 not found: ID does not exist"
Oct 29 23:21:48.059383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059351    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-socket\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.059559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059418    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-data\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.059559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059506    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-envoy\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.059653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059594    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-certs\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.059700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059664    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-socket" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.059780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059733    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-data" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.059843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059781    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8972826-8c7b-466f-9b19-818336fde77c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8972826-8c7b-466f-9b19-818336fde77c-istio-podinfo\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.059843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059825    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-credential-socket\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.059951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059866    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8972826-8c7b-466f-9b19-818336fde77c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8972826-8c7b-466f-9b19-818336fde77c-istiod-ca-cert\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.059951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059887    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-certs" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.060064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.059941    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-istio-token\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-istio-token\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.060064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.060010    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-kube-api-access-lsssd\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-kube-api-access-lsssd\") pod \"a8972826-8c7b-466f-9b19-818336fde77c\" (UID: \"a8972826-8c7b-466f-9b19-818336fde77c\") "
Oct 29 23:21:48.060165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.060107    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-credential-socket" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.060307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.060285    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8972826-8c7b-466f-9b19-818336fde77c-istiod-ca-cert" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:48.060992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.060966    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.061128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.060997    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.061128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.061025    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.061128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.061039    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.061128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.061052    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8972826-8c7b-466f-9b19-818336fde77c-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.061973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.061945    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-envoy" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.062439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.062139    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8972826-8c7b-466f-9b19-818336fde77c-istio-podinfo" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:48.062439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.062161    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-istio-token" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:48.062439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.062270    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-kube-api-access-lsssd" pod "a8972826-8c7b-466f-9b19-818336fde77c" (UID: "a8972826-8c7b-466f-9b19-818336fde77c"). InnerVolumeSpecName "kube-api-access-lsssd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:48.162708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.162679    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8972826-8c7b-466f-9b19-818336fde77c-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.162708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.162706    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.162708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.162715    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsssd\" (UniqueName: \"kubernetes.io/projected/a8972826-8c7b-466f-9b19-818336fde77c-kube-api-access-lsssd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.162987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.162724    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8972826-8c7b-466f-9b19-818336fde77c-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.267376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.267345    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"]
Oct 29 23:21:48.276649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.276626    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-c4dlf"]
Oct 29 23:21:48.491138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.491055    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8972826-8c7b-466f-9b19-818336fde77c" path="/var/lib/kubelet/pods/a8972826-8c7b-466f-9b19-818336fde77c/volumes"
Oct 29 23:21:48.935573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.935542    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"]
Oct 29 23:21:48.936030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:48.935828    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy" containerID="cri-o://ab5627e2ad70b44e676bc83476fef547ead438b83d70183cfccb739517d49fd3" gracePeriod=30
Oct 29 23:21:50.852411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:50.852377    4827 scope.go:122] "RemoveContainer" containerID="6660836bafa8a7981ef7021d407a148af0ee954d9521bef90cfe30901e408ee1"
Oct 29 23:21:50.862893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:50.862870    4827 scope.go:122] "RemoveContainer" containerID="b1104cabeb60579e8a4fce01d2a7518e83f9e6399185559daa9409b77d5733ec"
Oct 29 23:21:53.232791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:53.231822    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"]
Oct 29 23:21:53.232791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:53.232086    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" containerID="cri-o://34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce" gracePeriod=30
Oct 29 23:21:53.835788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:53.835620    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"]
Oct 29 23:21:53.846790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:53.836464    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy" containerID="cri-o://83e0fec097c94f67f7850078e4906476e603aa090327ba83c4cddc9c03b3ce79" gracePeriod=30
Oct 29 23:21:54.076791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.054546    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.114531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.082860    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" probeResult="failure" output="Get \"https://10.134.1.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.144277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.142876    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.257790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.252225    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.357651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.354475    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.435908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.435863    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.438744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.436321    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:21:54.478795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.473875    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.764000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.759890    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:54.883808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:54.871903    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:55.117845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.108238    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 29 23:21:55.118227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.118193    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:55.119101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.119079    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:21:55.126803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.126169    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" probeResult="failure" output="Get \"https://10.134.1.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:55.126803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.126478    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:21:55.143428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.143391    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:55.150389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.150362    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:21:55.376883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.375907    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:55.394924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.394877    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:21:55.405813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.398877    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:21:55.426129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.426074    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:55.490219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.478830    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:21:55.490219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.480294    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:21:55.490219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.481460    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:21:55.490219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.481520    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:21:55.490219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.483972    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:21:55.508609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.508584    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:21:55.718079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.718047    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" event={"ID":"b73447f3-09ac-4b57-8704-e46f8713987d","Type":"ContainerDied","Data":"ab5627e2ad70b44e676bc83476fef547ead438b83d70183cfccb739517d49fd3"}
Oct 29 23:21:55.718347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.718328    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:21:55.718591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.718536    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:21:55.718740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.718729    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:21:55.718884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.718872    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:21:55.719004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.718992    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:21:55.719136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.719125    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:21:55.719249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.678816    4827 generic.go:356] "Generic (PLEG): container finished" podID="b73447f3-09ac-4b57-8704-e46f8713987d" containerID="ab5627e2ad70b44e676bc83476fef547ead438b83d70183cfccb739517d49fd3" exitCode=0
Oct 29 23:21:55.720486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.720467    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:21:55.728796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.721013    4827 scope.go:122] "RemoveContainer" containerID="47f0cbc9ded3d9cba64bf50892d51ded48ac300df358da7ac4bbe35081439408"
Oct 29 23:21:55.765843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.765816    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:21:55.885591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.885557    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-socket\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.885799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.885683    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-kube-api-access-cq8th\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-kube-api-access-cq8th\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.885799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.885746    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73447f3-09ac-4b57-8704-e46f8713987d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73447f3-09ac-4b57-8704-e46f8713987d-istiod-ca-cert\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.885930 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.885814    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-istio-token\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-istio-token\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.885930 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.885861    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-socket" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:55.885930 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.885925    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-data\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.886089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.885982    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-envoy\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.886089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.886030    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-credential-socket\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.886089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.886060    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-certs\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.886245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.886175    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73447f3-09ac-4b57-8704-e46f8713987d-istiod-ca-cert" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:55.886303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.886248    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b73447f3-09ac-4b57-8704-e46f8713987d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73447f3-09ac-4b57-8704-e46f8713987d-istio-podinfo\") pod \"b73447f3-09ac-4b57-8704-e46f8713987d\" (UID: \"b73447f3-09ac-4b57-8704-e46f8713987d\") "
Oct 29 23:21:55.886303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.886282    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-data" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:55.886418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.886391    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-credential-socket" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:55.886641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.886618    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-certs" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:55.887374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.887348    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.887493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.887378    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73447f3-09ac-4b57-8704-e46f8713987d-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.887493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.887393    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.887493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.887407    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.887493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.887420    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.888514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.888469    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-istio-token" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:55.888702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.888684    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-envoy" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:55.889089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.889065    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-kube-api-access-cq8th" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "kube-api-access-cq8th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:55.889967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.889946    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b73447f3-09ac-4b57-8704-e46f8713987d-istio-podinfo" pod "b73447f3-09ac-4b57-8704-e46f8713987d" (UID: "b73447f3-09ac-4b57-8704-e46f8713987d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:55.989883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.989760    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73447f3-09ac-4b57-8704-e46f8713987d-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.989883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.989828    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73447f3-09ac-4b57-8704-e46f8713987d-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.989883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.989838    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq8th\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-kube-api-access-cq8th\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:55.989883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:55.989849    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73447f3-09ac-4b57-8704-e46f8713987d-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:56.557024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:56.556984    4827 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:21:56.557402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:56.557055    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:21:56.733123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:56.733084    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz" event={"ID":"b73447f3-09ac-4b57-8704-e46f8713987d","Type":"ContainerDied","Data":"d4c364c795591b1eef34c3d3eb57c47a63fc7622895fd1ac16042174914c85d4"}
Oct 29 23:21:56.733315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:56.733146    4827 scope.go:122] "RemoveContainer" containerID="ab5627e2ad70b44e676bc83476fef547ead438b83d70183cfccb739517d49fd3"
Oct 29 23:21:56.733315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:56.733103    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"
Oct 29 23:21:56.847058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:56.846981    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"]
Oct 29 23:21:56.858148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:56.858121    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-92svz"]
Oct 29 23:21:58.495177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.495141    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" path="/var/lib/kubelet/pods/b73447f3-09ac-4b57-8704-e46f8713987d/volumes"
Oct 29 23:21:58.545805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.545755    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:21:58.651216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651186    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-kube-api-access-ltv8v\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-kube-api-access-ltv8v\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651278    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-socket\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651313    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-data\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651423    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-envoy\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651506    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a092b8-5a06-4af1-9db7-870948480836-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a092b8-5a06-4af1-9db7-870948480836-istio-podinfo\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651577    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a092b8-5a06-4af1-9db7-870948480836-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a092b8-5a06-4af1-9db7-870948480836-istiod-ca-cert\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651587    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-socket" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:58.651674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651625    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-istio-token\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651722    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-certs\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.651853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.651753    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-credential-socket\") pod \"b8a092b8-5a06-4af1-9db7-870948480836\" (UID: \"b8a092b8-5a06-4af1-9db7-870948480836\") "
Oct 29 23:21:58.652400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.652341    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-data" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:58.652582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.652563    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-credential-socket" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:58.652663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.652587    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.652663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.652609    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.652820 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.652801    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-certs" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:58.653105 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.653085    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a092b8-5a06-4af1-9db7-870948480836-istiod-ca-cert" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:58.654858 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.654822    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-istio-token" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:58.660658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.660629    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-envoy" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:58.660888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.660755    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-kube-api-access-ltv8v" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "kube-api-access-ltv8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:58.662207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.662184    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a092b8-5a06-4af1-9db7-870948480836-istio-podinfo" pod "b8a092b8-5a06-4af1-9db7-870948480836" (UID: "b8a092b8-5a06-4af1-9db7-870948480836"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:58.753238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.753204    4827 generic.go:356] "Generic (PLEG): container finished" podID="b8a092b8-5a06-4af1-9db7-870948480836" containerID="34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce" exitCode=0
Oct 29 23:21:58.753425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.753287    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"
Oct 29 23:21:58.753425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.753286    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" event={"ID":"b8a092b8-5a06-4af1-9db7-870948480836","Type":"ContainerDied","Data":"34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce"}
Oct 29 23:21:58.753589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.753449    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh" event={"ID":"b8a092b8-5a06-4af1-9db7-870948480836","Type":"ContainerDied","Data":"dae00e199aaae9846dab0958a7330ccf64dc1b3fdc3358f350640b2d7322d5f8"}
Oct 29 23:21:58.753589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.753479    4827 scope.go:122] "RemoveContainer" containerID="34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce"
Oct 29 23:21:58.754206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.754179    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltv8v\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-kube-api-access-ltv8v\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.754206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.754209    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.754408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.754223    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a092b8-5a06-4af1-9db7-870948480836-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.754408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.754236    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a092b8-5a06-4af1-9db7-870948480836-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.754408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.754250    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a092b8-5a06-4af1-9db7-870948480836-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.754408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.754263    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.754408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.754276    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a092b8-5a06-4af1-9db7-870948480836-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:58.765997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.765975    4827 scope.go:122] "RemoveContainer" containerID="34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce"
Oct 29 23:21:58.766268 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:21:58.766249    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce\": container with ID starting with 34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce not found: ID does not exist" containerID="34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce"
Oct 29 23:21:58.766342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.766281    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce"} err="failed to get container status \"34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce\": rpc error: code = NotFound desc = could not find container \"34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce\": container with ID starting with 34c96acbc079a0daf0f9ec97beea1b665ae710a11d6847e3bbf963ce2af730ce not found: ID does not exist"
Oct 29 23:21:58.817006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.816967    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"]
Oct 29 23:21:58.821017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:58.820992    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lvvxh"]
Oct 29 23:21:59.765577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.765544    4827 generic.go:356] "Generic (PLEG): container finished" podID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerID="83e0fec097c94f67f7850078e4906476e603aa090327ba83c4cddc9c03b3ce79" exitCode=0
Oct 29 23:21:59.766101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.765617    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" event={"ID":"b3b00b28-26d5-4247-871e-1fc3bbe8f020","Type":"ContainerDied","Data":"83e0fec097c94f67f7850078e4906476e603aa090327ba83c4cddc9c03b3ce79"}
Oct 29 23:21:59.805852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.805825    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:21:59.899584 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.899551    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-certs\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.900081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.900056    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-envoy\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.900294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.900223    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-token\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-token\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.900496 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.900465    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istiod-ca-cert\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.900624 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.900590    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-credential-socket\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.902051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.900786    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-kube-api-access-wh548\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-kube-api-access-wh548\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.902051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.901021    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-podinfo\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.902051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.901055    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-socket\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.902051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.901347    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-socket" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:59.902051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.901590    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-certs" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:59.902051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.901888    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-data\") pod \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\" (UID: \"b3b00b28-26d5-4247-871e-1fc3bbe8f020\") "
Oct 29 23:21:59.902433 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.902105    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-credential-socket" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:59.902817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.902733    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-data" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:59.902920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.902886    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istiod-ca-cert" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.905459    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-podinfo" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.905690    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.905709    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.905723    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.905736    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.905749    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.905782    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:21:59.907374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.906101    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-envoy" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:59.907925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.907899    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-token" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:59.909184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:21:59.909159    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-kube-api-access-wh548" pod "b3b00b28-26d5-4247-871e-1fc3bbe8f020" (UID: "b3b00b28-26d5-4247-871e-1fc3bbe8f020"). InnerVolumeSpecName "kube-api-access-wh548". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:22:00.007917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.007887    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:22:00.007917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.007913    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:22:00.007917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.007924    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh548\" (UniqueName: \"kubernetes.io/projected/b3b00b28-26d5-4247-871e-1fc3bbe8f020-kube-api-access-wh548\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:22:00.490780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.490736    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a092b8-5a06-4af1-9db7-870948480836" path="/var/lib/kubelet/pods/b8a092b8-5a06-4af1-9db7-870948480836/volumes"
Oct 29 23:22:00.776497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.776420    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"
Oct 29 23:22:00.776497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.776449    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc" event={"ID":"b3b00b28-26d5-4247-871e-1fc3bbe8f020","Type":"ContainerDied","Data":"3df909c4a42b3cdc5b7a751deb6d430441965a3b420d55b99dc3711dce9fddea"}
Oct 29 23:22:00.777036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.776511    4827 scope.go:122] "RemoveContainer" containerID="83e0fec097c94f67f7850078e4906476e603aa090327ba83c4cddc9c03b3ce79"
Oct 29 23:22:00.870027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.838285    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"]
Oct 29 23:22:00.969756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:00.969721    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-gg4kc"]
Oct 29 23:22:02.491570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:02.491534    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" path="/var/lib/kubelet/pods/b3b00b28-26d5-4247-871e-1fc3bbe8f020/volumes"
Oct 29 23:22:02.826496 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:22:02.826415    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:22:10.705708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.705667    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"]
Oct 29 23:22:10.706952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.706926    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" containerName="maas-api"
Oct 29 23:22:10.706952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.706952    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" containerName="maas-api"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.706981    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.706990    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707008    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707015    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707048    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707055    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707072    4827 cpu_manager.go:414] "containerMap: removing container" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707080    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707102    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707109    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707119    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707130    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707156    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy"
Oct 29 23:22:10.707160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707163    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy"
Oct 29 23:22:10.707924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707324    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b00b28-26d5-4247-871e-1fc3bbe8f020" containerName="istio-proxy"
Oct 29 23:22:10.707924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707339    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="299221fe-bd31-4270-9adb-ff8eaee70ed6" containerName="istio-proxy"
Oct 29 23:22:10.707924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707351    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy"
Oct 29 23:22:10.707924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707361    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8972826-8c7b-466f-9b19-818336fde77c" containerName="istio-proxy"
Oct 29 23:22:10.707924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707374    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a092b8-5a06-4af1-9db7-870948480836" containerName="istio-proxy"
Oct 29 23:22:10.707924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707388    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa4f7dc-0517-47d3-8a31-b424b9cf7bbe" containerName="maas-api"
Oct 29 23:22:10.707924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.707400    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb1ccfd-7044-4b45-8e8a-4da139673db6" containerName="istio-proxy"
Oct 29 23:22:10.712133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.712110    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.733750 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.733717    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"]
Oct 29 23:22:10.741419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.741346    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a37090\"/\"default-dockercfg-dxmm9\""
Oct 29 23:22:10.741419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.741404    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a37090\"/\"test-model-a37090-kserve-self-signed-certs\""
Oct 29 23:22:10.741633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.741402    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a37090\"/\"openshift-service-ca.crt\""
Oct 29 23:22:10.741633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.741423    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a37090\"/\"kube-root-ca.crt\""
Oct 29 23:22:10.801912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.801873    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-home\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.802084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.802011    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-dshm\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.802170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.802153    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wkb\" (UniqueName: \"kubernetes.io/projected/afd19809-a784-4509-bdf4-50c8049f76e9-kube-api-access-x8wkb\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.802233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.802203    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-tmp-dir\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.802233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.802228    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-model-cache\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.802358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.802293    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd19809-a784-4509-bdf4-50c8049f76e9-tls-certs\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.903547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.903509    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-dshm\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.903803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.903614    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wkb\" (UniqueName: \"kubernetes.io/projected/afd19809-a784-4509-bdf4-50c8049f76e9-kube-api-access-x8wkb\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.903803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.903678    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-tmp-dir\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.903803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.903708    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-model-cache\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.903803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.903753    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd19809-a784-4509-bdf4-50c8049f76e9-tls-certs\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.904066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.904046    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-home\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.904468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.904444    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-home\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.905117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.905081    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-model-cache\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.905323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.905305    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-tmp-dir\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.906963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.906937    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-dshm\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.907988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.907967    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd19809-a784-4509-bdf4-50c8049f76e9-tls-certs\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:10.926466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:10.926441    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wkb\" (UniqueName: \"kubernetes.io/projected/afd19809-a784-4509-bdf4-50c8049f76e9-kube-api-access-x8wkb\") pod \"test-model-a37090-kserve-6cfb57c7dc-lth4p\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") " pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:11.040852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:11.038602    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:11.468644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:11.468614    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"]
Oct 29 23:22:11.469436 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:22:11.469407    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafd19809_a784_4509_bdf4_50c8049f76e9.slice/crio-bdecda6e4b8ca95be2f3b6283f806dc4613197ad027014eb4f062d487b53f78d WatchSource:0}: Error finding container bdecda6e4b8ca95be2f3b6283f806dc4613197ad027014eb4f062d487b53f78d: Status 404 returned error can't find the container with id bdecda6e4b8ca95be2f3b6283f806dc4613197ad027014eb4f062d487b53f78d
Oct 29 23:22:12.053010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:12.052970    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" event={"ID":"afd19809-a784-4509-bdf4-50c8049f76e9","Type":"ContainerStarted","Data":"97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d"}
Oct 29 23:22:12.053010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:12.053022    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" event={"ID":"afd19809-a784-4509-bdf4-50c8049f76e9","Type":"ContainerStarted","Data":"bdecda6e4b8ca95be2f3b6283f806dc4613197ad027014eb4f062d487b53f78d"}
Oct 29 23:22:12.112967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:12.112906    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" podStartSLOduration=2.112877259 podStartE2EDuration="2.112877259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:12.108156218 +0000 UTC m=+2367.731067239" watchObservedRunningTime="2026-10-29 23:22:12.112877259 +0000 UTC m=+2367.735788281"
Oct 29 23:22:13.039541 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:13.039502    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:15.147068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:15.147037    4827 generic.go:356] "Generic (PLEG): container finished" podID="6c041e2d-bd10-460e-a154-06946afd4220" containerID="b04db3035bfab780619b1bbacdd4b8fe1ff9b78252b07a8160be3947af93996a" exitCode=0
Oct 29 23:22:15.147513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:15.147103    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" event={"ID":"6c041e2d-bd10-460e-a154-06946afd4220","Type":"ContainerDied","Data":"b04db3035bfab780619b1bbacdd4b8fe1ff9b78252b07a8160be3947af93996a"}
Oct 29 23:22:15.147609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:15.147594    4827 scope.go:122] "RemoveContainer" containerID="b04db3035bfab780619b1bbacdd4b8fe1ff9b78252b07a8160be3947af93996a"
Oct 29 23:22:15.149892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:15.149858    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:15.150026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:15.149928    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:15.150026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:15.149962    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:16.156286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:16.156255    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" event={"ID":"6c041e2d-bd10-460e-a154-06946afd4220","Type":"ContainerStarted","Data":"b88e3ad478fe8a50b38795e5973c45130d901abfbf1f392e1aabdfc6ae751180"}
Oct 29 23:22:16.602266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:16.602176    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:22:16.605466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:16.605430    4827 patch_prober.go:29] interesting pod/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 29 23:22:16.605631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:16.605509    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 29 23:22:17.106370 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.106335    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"]
Oct 29 23:22:17.107440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.107414    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73447f3-09ac-4b57-8704-e46f8713987d" containerName="istio-proxy"
Oct 29 23:22:17.120630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.111803    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.137616    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.137670    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.137706    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.137732    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.137856    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7df262-3b46-4141-a339-bea7561b622f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.138006    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7df262-3b46-4141-a339-bea7561b622f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.138084    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.138352    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.140126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.138493    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw6r2\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-kube-api-access-bw6r2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.150462 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.150433    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"]
Oct 29 23:22:17.223796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.222913    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"]
Oct 29 23:22:17.223796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.222950    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"]
Oct 29 23:22:17.223796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.223026    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.241009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.240979    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.241366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.241341    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.241535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.241517    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.241712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.241698    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw6r2\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-kube-api-access-bw6r2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.241857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.241822    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.241958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.241941    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.242065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.242053    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.242154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.242144    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.242226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.242215    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.242324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.242309    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7df262-3b46-4141-a339-bea7561b622f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.242419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.242410    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7df262-3b46-4141-a339-bea7561b622f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.242545 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.242455    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.242637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.242614    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.243114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.243096    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7df262-3b46-4141-a339-bea7561b622f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.247805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.244812    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.247805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.245030    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7df262-3b46-4141-a339-bea7561b622f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.281813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.265039    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw6r2\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-kube-api-access-bw6r2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.281813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.279730    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n6pr5\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347231    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347294    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jvd\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-kube-api-access-m9jvd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347343    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347422    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347461    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347552    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347626    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347850    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.350792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.347883    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e63f57-51c7-426a-8967-3f8e1130f56d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450087    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450140    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e63f57-51c7-426a-8967-3f8e1130f56d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450349    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450386    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jvd\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-kube-api-access-m9jvd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450439    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450514    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450555    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450641    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.450705    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.451365    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.451614    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.454163    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e63f57-51c7-426a-8967-3f8e1130f56d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.454788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.454436    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.462256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.462193    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.497861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.494760    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.497861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.496532    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jvd\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-kube-api-access-m9jvd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.499577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.499548    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.501555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.501531    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k2lxp\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.517279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.517259    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:17.541434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.541409    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:17.562604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.562581    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:22:17.602440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.602403    4827 patch_prober.go:29] interesting pod/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 29 23:22:17.602629 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.602462    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 29 23:22:17.893323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.893293    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"]
Oct 29 23:22:17.894737 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:22:17.894699    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7df262_3b46_4141_a339_bea7561b622f.slice/crio-02d26eadbad350597b53939ea20087e50677155904d46bceabbb84d7731461fe WatchSource:0}: Error finding container 02d26eadbad350597b53939ea20087e50677155904d46bceabbb84d7731461fe: Status 404 returned error can't find the container with id 02d26eadbad350597b53939ea20087e50677155904d46bceabbb84d7731461fe
Oct 29 23:22:17.897981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.897948    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:17.898050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.898022    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:17.898091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.898068    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:17.917255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.917232    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"]
Oct 29 23:22:17.921264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.921219    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:17.921420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.921328    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:17.921420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:17.921374    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:18.216062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.215955    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" event={"ID":"59e63f57-51c7-426a-8967-3f8e1130f56d","Type":"ContainerStarted","Data":"6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151"}
Oct 29 23:22:18.216062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.216017    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" event={"ID":"59e63f57-51c7-426a-8967-3f8e1130f56d","Type":"ContainerStarted","Data":"59b103b1a1d05777ecd964a50d1a0e8364adb79c3ec08113bc7fbffa1a41f21f"}
Oct 29 23:22:18.219040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.218389    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" event={"ID":"ff7df262-3b46-4141-a339-bea7561b622f","Type":"ContainerStarted","Data":"3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c"}
Oct 29 23:22:18.219040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.218603    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" event={"ID":"ff7df262-3b46-4141-a339-bea7561b622f","Type":"ContainerStarted","Data":"02d26eadbad350597b53939ea20087e50677155904d46bceabbb84d7731461fe"}
Oct 29 23:22:18.263848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.261874    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podStartSLOduration=1.261851926 podStartE2EDuration="1.261851926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:18.259421373 +0000 UTC m=+2373.882332395" watchObservedRunningTime="2026-10-29 23:22:18.261851926 +0000 UTC m=+2373.884762949"
Oct 29 23:22:18.310793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.310398    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podStartSLOduration=1.310380213 podStartE2EDuration="1.310380213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:18.307256712 +0000 UTC m=+2373.930167767" watchObservedRunningTime="2026-10-29 23:22:18.310380213 +0000 UTC m=+2373.933291234"
Oct 29 23:22:18.517995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.517890    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:18.541622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.541595    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:18.602786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.602286    4827 patch_prober.go:29] interesting pod/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 29 23:22:18.602786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:18.602353    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 29 23:22:19.522018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:19.521959    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:19.584791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:19.549022    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:19.584791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:19.547293    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:19.584791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:19.549128    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:19.649796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:19.640352    4827 patch_prober.go:29] interesting pod/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 29 23:22:19.649796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:19.640416    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 29 23:22:20.527870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:20.527403    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:20.527870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:20.527463    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:20.545195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:20.544983    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:20.545195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:20.545046    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:20.613112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:20.613078    4827 patch_prober.go:29] interesting pod/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 29 23:22:20.613597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:20.613478    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 29 23:22:21.044175 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.044139    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:21.062380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.062345    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:21.349033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.348951    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:22:21.527784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.527211    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:21.527784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.527273    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:21.602844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.598552    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:21.602844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.598619    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:21.602844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.602139    4827 patch_prober.go:29] interesting pod/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 29 23:22:21.602844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:21.602193    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 29 23:22:22.522192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:22.522147    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:22.522450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:22.522421    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:22.542196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:22.542166    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:22.542560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:22.542218    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:22.624120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:22.622404    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:22:22.625982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:22.625958    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:22:23.522127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:23.522091    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:23.522460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:23.522431    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:23.544791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:23.544119    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:23.544791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:23.544170    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:24.517969 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:24.517926    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:24.518229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:24.518200    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:24.541980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:24.541953    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:24.542217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:24.542193    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:25.518574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:25.518539    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:25.519120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:25.518597    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:25.541875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:25.541842    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:25.542036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:25.541901    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:26.518668 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:26.518635    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n6pr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:22:26.519203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:26.518695    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:22:26.542538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:26.542503    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k2lxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 29 23:22:26.542701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:26.542562    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 29 23:22:27.915411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:27.915336    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:27.917044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:27.917017    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:27.918092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:27.918069    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:22:27.963181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:27.963154    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:27.963810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:27.963787    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:27.964636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:27.964612    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:22:32.162046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.162013    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"]
Oct 29 23:22:32.167886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.167867    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.189454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189422    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.189612 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189567    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.189690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189637    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.189690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189677    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.189867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189751    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvx2\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-kube-api-access-nvvx2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.189867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189825    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.189867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189851    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.190039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.189974    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.190039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.190004    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.191291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.191269    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"]
Oct 29 23:22:32.292062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292029    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292090    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292125    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292148    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292197    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvx2\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-kube-api-access-nvvx2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292217    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292238    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292351    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292389    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292501    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292549    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292598    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292822    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.292993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.292965    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.294954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.294899    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.295135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.295113    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.314742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.314714    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.315032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.315015    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvx2\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-kube-api-access-nvvx2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8q72b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.484678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.484601    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:32.811429 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:22:32.811395    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa6ad85_8294_49a5_8c24_7d4fb9e1582b.slice/crio-ecd7f56fda81e80b599b45b2ca6575390ce155aad2cb97703f9b90b65d198290 WatchSource:0}: Error finding container ecd7f56fda81e80b599b45b2ca6575390ce155aad2cb97703f9b90b65d198290: Status 404 returned error can't find the container with id ecd7f56fda81e80b599b45b2ca6575390ce155aad2cb97703f9b90b65d198290
Oct 29 23:22:32.812614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.812590    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"]
Oct 29 23:22:32.816296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.816257    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:32.816405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.816350    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:32.816405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:32.816392    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:22:32.830427 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:22:32.830392    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:22:33.457153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:33.457114    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" event={"ID":"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b","Type":"ContainerStarted","Data":"e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8"}
Oct 29 23:22:33.457625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:33.457189    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" event={"ID":"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b","Type":"ContainerStarted","Data":"ecd7f56fda81e80b599b45b2ca6575390ce155aad2cb97703f9b90b65d198290"}
Oct 29 23:22:33.485827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:33.485797    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:33.487383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:33.487359    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:33.487508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:33.487401    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:33.516106 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:33.516049    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podStartSLOduration=1.5160301999999999 podStartE2EDuration="1.5160302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:33.51422736 +0000 UTC m=+2389.137138382" watchObservedRunningTime="2026-10-29 23:22:33.5160302 +0000 UTC m=+2389.138941221"
Oct 29 23:22:34.494598 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:34.494554    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:34.495145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:34.494621    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:35.488137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:35.488107    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:35.488347 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:35.488216    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:36.485222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:36.485184    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:36.487913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:36.487867    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:37.487792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:37.486341    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:37.487792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:37.486399    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:38.485194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:38.485156    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:38.485513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:38.485487    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:39.485580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:39.485547    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:39.486115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:39.485618    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:40.485689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:40.485649    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8q72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Oct 29 23:22:40.486142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:40.485715    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Oct 29 23:22:41.930831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:41.930800    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:41.931669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:41.931309    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:41.932541 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:41.932518    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:22:42.195827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.195727    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"]
Oct 29 23:22:42.202144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.202113    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.219045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.219004    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-024aa6\"/\"default-dockercfg-45n7t\""
Oct 29 23:22:42.222568 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.222546    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-024aa6\"/\"test-model-024aa6-kserve-self-signed-certs\""
Oct 29 23:22:42.222822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.222759    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"]
Oct 29 23:22:42.223618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.223592    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-024aa6\"/\"openshift-service-ca.crt\""
Oct 29 23:22:42.223720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.223627    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-024aa6\"/\"kube-root-ca.crt\""
Oct 29 23:22:42.278441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.278363    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e24bb-c63d-4658-88fd-14ca4585b952-tls-certs\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.278646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.278607    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-model-cache\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.278813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.278747    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-tmp-dir\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.278973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.278849    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-home\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.278973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.278888    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-dshm\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.279109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.279025    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g756d\" (UniqueName: \"kubernetes.io/projected/df8e24bb-c63d-4658-88fd-14ca4585b952-kube-api-access-g756d\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381256    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e24bb-c63d-4658-88fd-14ca4585b952-tls-certs\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381393    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-model-cache\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381465    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-tmp-dir\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381501    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-home\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381531    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-dshm\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381591    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g756d\" (UniqueName: \"kubernetes.io/projected/df8e24bb-c63d-4658-88fd-14ca4585b952-kube-api-access-g756d\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381850    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-tmp-dir\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.381942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381884    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-model-cache\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.382010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.381939    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-home\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.383731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.383705    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-dshm\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.383875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.383841    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e24bb-c63d-4658-88fd-14ca4585b952-tls-certs\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.403622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.403588    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g756d\" (UniqueName: \"kubernetes.io/projected/df8e24bb-c63d-4658-88fd-14ca4585b952-kube-api-access-g756d\") pod \"test-model-024aa6-kserve-98c965cd7-9tzrq\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") " pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:42.548840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:42.542448    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:43.094503 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:22:43.094442    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf8e24bb_c63d_4658_88fd_14ca4585b952.slice/crio-9811d42c908f5f783654d64166cf3b7a9e0267c02312b0c04cb8c8709e03cee3 WatchSource:0}: Error finding container 9811d42c908f5f783654d64166cf3b7a9e0267c02312b0c04cb8c8709e03cee3: Status 404 returned error can't find the container with id 9811d42c908f5f783654d64166cf3b7a9e0267c02312b0c04cb8c8709e03cee3
Oct 29 23:22:43.094909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:43.094604    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"]
Oct 29 23:22:43.674867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:43.674826    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" event={"ID":"df8e24bb-c63d-4658-88fd-14ca4585b952","Type":"ContainerStarted","Data":"eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675"}
Oct 29 23:22:43.675045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:43.674878    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" event={"ID":"df8e24bb-c63d-4658-88fd-14ca4585b952","Type":"ContainerStarted","Data":"9811d42c908f5f783654d64166cf3b7a9e0267c02312b0c04cb8c8709e03cee3"}
Oct 29 23:22:43.724327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:43.724265    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" podStartSLOduration=1.724246424 podStartE2EDuration="1.724246424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:43.717969294 +0000 UTC m=+2399.340880314" watchObservedRunningTime="2026-10-29 23:22:43.724246424 +0000 UTC m=+2399.347157451"
Oct 29 23:22:44.592796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:44.583247    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:47.281246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:47.281221    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:22:47.401056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:47.401031    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:22:52.542929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:52.542888    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:52.557286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:52.557259    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:52.882677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:52.882652    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:22:59.020789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:59.020726    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:22:59.166978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:59.166935    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:22:59.315523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:59.315421    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main" probeResult="failure" output="Get \"https://10.134.1.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:22:59.425598 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:22:59.425561    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:01.816792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:01.801547    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:01.883825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:01.883777    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerName="main" probeResult="failure" output="Get \"https://10.134.1.43:8000/ready\": context deadline exceeded"
Oct 29 23:23:02.045588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.045543    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main" probeResult="failure" output="Get \"https://10.134.1.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.113513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.113007    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.172961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.167897    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.255952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.255698    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 23:23:02.303305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.303260    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main" probeResult="failure" output="Get \"https://10.134.1.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.360967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.360051    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.415565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.414846    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.445032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.444992    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.460916 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.460873    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.499406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.499367    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": context deadline exceeded"
Oct 29 23:23:02.580180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.580138    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.630884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.630838    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" probeResult="failure" output="Get \"https://10.134.1.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.734849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.728032    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.807171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.807127    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:02.808345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.808320    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:23:02.897945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:02.897899    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" probeResult="failure" output="Get \"https://10.134.1.33:8000/health\": context deadline exceeded"
Oct 29 23:23:03.153799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.151889    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:23:03.301924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.207125    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:23:03.336832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.323857    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:03.336832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.326458    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.35:8000/health\": context deadline exceeded"
Oct 29 23:23:03.336832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.326964    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:23:03.336832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.333956    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 29 23:23:03.336832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.335992    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main" probeResult="failure" output="Get \"https://10.134.1.39:8000/ready\": context deadline exceeded"
Oct 29 23:23:03.337309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.336957    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:23:03.369841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.366298    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:23:03.369841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.366363    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:23:03.417529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.417428    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:03.418148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.418126    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:23:03.453788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.450005    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:23:03.475236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.475171    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:03.491584 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.491550    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:23:03.552843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.548145    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerName="main" probeResult="failure" output="Get \"https://10.134.1.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:03.635947 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:23:03.635824    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:23:03.637967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.637935    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" probeResult="failure" output="Get \"https://10.134.1.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:03.642686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.639807    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb"
Oct 29 23:23:03.642928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.642913    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:23:03.643071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.643061    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:23:03.646189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.645197    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:23:03.662007 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.661986    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:23:03.679927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.679850    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:23:03.700333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.699689    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:23:03.738378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:03.738353    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:23:08.611890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:08.611858    4827 patch_prober.go:29] interesting pod/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:23:08.612426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:08.611908    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:10.548258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:10.548223    4827 generic.go:356] "Generic (PLEG): container finished" podID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerID="1f409d85d3ffa9b8a286a6ca3c0bfd865fd46f45ae3c91c8f0034a90cd19d2dd" exitCode=0
Oct 29 23:23:10.548851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:10.548301    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" event={"ID":"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e","Type":"ContainerDied","Data":"1f409d85d3ffa9b8a286a6ca3c0bfd865fd46f45ae3c91c8f0034a90cd19d2dd"}
Oct 29 23:23:10.548969 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:10.548951    4827 scope.go:122] "RemoveContainer" containerID="1f409d85d3ffa9b8a286a6ca3c0bfd865fd46f45ae3c91c8f0034a90cd19d2dd"
Oct 29 23:23:10.551221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:10.551188    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:23:10.551355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:10.551276    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:23:10.551355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:10.551325    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:23:11.580313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:11.562905    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" event={"ID":"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e","Type":"ContainerStarted","Data":"b1ba41ce76fab7fde919b52d7d56a5b45fda6a25ccc2adc719ba055e4304bc28"}
Oct 29 23:23:11.714621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:11.714587    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:23:11.714887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:11.714755    4827 patch_prober.go:29] interesting pod/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:23:11.714887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:11.714848    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Oct 29 23:23:12.713962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:12.713907    4827 patch_prober.go:29] interesting pod/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:23:12.714435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:12.713973    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Oct 29 23:23:13.715821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:13.714296    4827 patch_prober.go:29] interesting pod/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:23:13.715821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:13.714364    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Oct 29 23:23:14.713921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:14.713889    4827 patch_prober.go:29] interesting pod/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:23:14.714146 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:14.713946    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Oct 29 23:23:15.714059 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:15.714028    4827 patch_prober.go:29] interesting pod/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:23:15.714463 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:15.714090    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Oct 29 23:23:16.714822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:16.714782    4827 patch_prober.go:29] interesting pod/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:23:16.715250 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:16.714833    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Oct 29 23:23:17.719374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:17.719346    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:23:17.719925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:17.719906    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:23:17.720919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:17.720896    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:23:32.825431 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:23:32.825378    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:23:35.361951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.361904    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:35.453964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.453908    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:35.455192 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.455159    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.35:8000/ready\": context deadline exceeded"
Oct 29 23:23:35.478165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.478129    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"]
Oct 29 23:23:35.478681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.478632    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy" containerID="cri-o://d1463cac3feb50d4c67765aac7ccef3d91a194b07c2f23bf424549fd23bbd714" gracePeriod=30
Oct 29 23:23:35.689821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.689464    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:35.692294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.692254    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:23:35.701329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.700709    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:35.716784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.714894    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 29 23:23:35.716784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.715249    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:23:35.716784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.715533    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:23:35.735210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.719492    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main" probeResult="failure" output="Get \"https://10.134.1.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:35.739888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.739838    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:35.760810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:35.759010    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:23:36.043284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:36.043207    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"]
Oct 29 23:23:39.299921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:39.299889    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"]
Oct 29 23:23:40.971308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:40.971265    4827 generic.go:356] "Generic (PLEG): container finished" podID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerID="d1463cac3feb50d4c67765aac7ccef3d91a194b07c2f23bf424549fd23bbd714" exitCode=0
Oct 29 23:23:40.971856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:40.971328    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" event={"ID":"772c47d7-cc86-4369-9921-7fbe19d2a3bd","Type":"ContainerDied","Data":"d1463cac3feb50d4c67765aac7ccef3d91a194b07c2f23bf424549fd23bbd714"}
Oct 29 23:23:40.971856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:40.971381    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x" event={"ID":"772c47d7-cc86-4369-9921-7fbe19d2a3bd","Type":"ContainerDied","Data":"92cd7f6f0fddaf9659e173ee8a391f7b9f452690977d955f6293335b5a314594"}
Oct 29 23:23:40.971856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:40.971404    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92cd7f6f0fddaf9659e173ee8a391f7b9f452690977d955f6293335b5a314594"
Oct 29 23:23:40.995325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:40.995301    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:23:41.079751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.079659    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istiod-ca-cert\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.079956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.079751    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-certs\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.079956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.079880    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-credential-socket\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.080095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080007    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-token\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-token\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.080095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080076    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-data\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.080221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080127    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-certs" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:41.080221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080164    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istiod-ca-cert" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:23:41.080353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080217    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-credential-socket" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:41.080353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080248    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-socket\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.080353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080294    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-podinfo\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.080530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080418    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-data" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:41.080530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080434    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-envoy\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.080530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080448    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-socket" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:41.080530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.080466    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-kube-api-access-bqbvn\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-kube-api-access-bqbvn\") pod \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\" (UID: \"772c47d7-cc86-4369-9921-7fbe19d2a3bd\") "
Oct 29 23:23:41.081401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.081370    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.081549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.081404    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.081549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.081419    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.081549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.081432    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.081549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.081444    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.082811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.082681    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-envoy" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:41.082811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.082758    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-podinfo" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:23:41.082980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.082840    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-token" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:23:41.082980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.082942    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-kube-api-access-bqbvn" pod "772c47d7-cc86-4369-9921-7fbe19d2a3bd" (UID: "772c47d7-cc86-4369-9921-7fbe19d2a3bd"). InnerVolumeSpecName "kube-api-access-bqbvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:23:41.183036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.183007    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.183036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.183033    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.183036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.183042    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772c47d7-cc86-4369-9921-7fbe19d2a3bd-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.183290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.183051    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqbvn\" (UniqueName: \"kubernetes.io/projected/772c47d7-cc86-4369-9921-7fbe19d2a3bd-kube-api-access-bqbvn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.979844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:41.979812    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"
Oct 29 23:23:42.041818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:42.041785    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"]
Oct 29 23:23:42.052502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:42.052479    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-k5g5x"]
Oct 29 23:23:42.499960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:42.499922    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" path="/var/lib/kubelet/pods/772c47d7-cc86-4369-9921-7fbe19d2a3bd/volumes"
Oct 29 23:23:46.907945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:46.907912    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"]
Oct 29 23:23:50.956837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:50.956807    4827 scope.go:122] "RemoveContainer" containerID="d1463cac3feb50d4c67765aac7ccef3d91a194b07c2f23bf424549fd23bbd714"
Oct 29 23:23:51.085592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.085554    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main" containerID="cri-o://b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd" gracePeriod=45
Oct 29 23:23:51.377196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.377170    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:23:51.400376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.400347    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-kube-api-access-xgdjq\" (UniqueName: \"kubernetes.io/projected/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-kube-api-access-xgdjq\") pod \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") "
Oct 29 23:23:51.400616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.400594    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tls-certs\") pod \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") "
Oct 29 23:23:51.400678 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.400664    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-home\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-home\") pod \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") "
Oct 29 23:23:51.400747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.400714    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-dshm\") pod \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") "
Oct 29 23:23:51.401027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.400874    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-model-cache\") pod \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") "
Oct 29 23:23:51.401027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.400944    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-home" pod "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" (UID: "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:51.401179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.401104    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-model-cache" pod "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" (UID: "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:51.401179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.401155    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tmp-dir\") pod \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\" (UID: \"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5\") "
Oct 29 23:23:51.401448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.401423    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tmp-dir" pod "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" (UID: "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:51.402039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.402017    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:51.402097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.402046    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:51.402097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.402083    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:51.403075 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.403048    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-dshm" pod "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" (UID: "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:51.403189 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.403169    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tls-certs" pod "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" (UID: "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:23:51.403248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.403218    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-kube-api-access-xgdjq" pod "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" (UID: "962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5"). InnerVolumeSpecName "kube-api-access-xgdjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:23:51.503183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.503151    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:51.503183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.503178    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgdjq\" (UniqueName: \"kubernetes.io/projected/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-kube-api-access-xgdjq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:51.503183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.503190    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:51.938487 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:51.936420    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" podUID="2063e478-02e9-4920-a461-50223a414968" containerName="payload-pre-processing" containerID="cri-o://43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b" gracePeriod=25
Oct 29 23:23:52.069386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.069356    4827 generic.go:356] "Generic (PLEG): container finished" podID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerID="b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd" exitCode=0
Oct 29 23:23:52.069883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.069400    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"
Oct 29 23:23:52.069883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.069436    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" event={"ID":"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5","Type":"ContainerDied","Data":"b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd"}
Oct 29 23:23:52.069883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.069475    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c" event={"ID":"962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5","Type":"ContainerDied","Data":"940001d7a8a14bf8496e4240363cd45a4b54ba8f1bc7bb383ce62763887a3b6e"}
Oct 29 23:23:52.069883 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.069499    4827 scope.go:122] "RemoveContainer" containerID="b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd"
Oct 29 23:23:52.084607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.084587    4827 scope.go:122] "RemoveContainer" containerID="b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd"
Oct 29 23:23:52.084982 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:23:52.084961    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd\": container with ID starting with b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd not found: ID does not exist" containerID="b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd"
Oct 29 23:23:52.085062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.084990    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd"} err="failed to get container status \"b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd\": rpc error: code = NotFound desc = could not find container \"b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd\": container with ID starting with b15f0c46647e2436f1ff0dd1041d323ac325d352dca11bc5cc27ec02f463d4fd not found: ID does not exist"
Oct 29 23:23:52.121710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.121665    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"]
Oct 29 23:23:52.131636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.131604    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f4ce70/test-model-f4ce70-kserve-6b8cbf9f7c-6d76c"]
Oct 29 23:23:52.492214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:52.492176    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" path="/var/lib/kubelet/pods/962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5/volumes"
Oct 29 23:23:54.280305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.280255    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" podUID="2063e478-02e9-4920-a461-50223a414968" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:23:54.323052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.323016    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main" containerID="cri-o://7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b" gracePeriod=45
Oct 29 23:23:54.450332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.450292    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.35:8000/ready\": dial tcp 10.134.1.35:8000: connect: connection refused"
Oct 29 23:23:54.517382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.513025    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"]
Oct 29 23:23:54.886514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.886487    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:23:54.969929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.969900    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-tmp-dir\") pod \"fc6e0560-7627-4f0d-981e-e607a2d5598d\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") "
Oct 29 23:23:54.970094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.970033    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-home\") pod \"fc6e0560-7627-4f0d-981e-e607a2d5598d\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") "
Oct 29 23:23:54.970163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.970142    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-dshm\") pod \"fc6e0560-7627-4f0d-981e-e607a2d5598d\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") "
Oct 29 23:23:54.970232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.970212    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc6e0560-7627-4f0d-981e-e607a2d5598d-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e0560-7627-4f0d-981e-e607a2d5598d-tls-certs\") pod \"fc6e0560-7627-4f0d-981e-e607a2d5598d\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") "
Oct 29 23:23:54.970450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.970427    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6e0560-7627-4f0d-981e-e607a2d5598d-kube-api-access-cthhc\" (UniqueName: \"kubernetes.io/projected/fc6e0560-7627-4f0d-981e-e607a2d5598d-kube-api-access-cthhc\") pod \"fc6e0560-7627-4f0d-981e-e607a2d5598d\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") "
Oct 29 23:23:54.970538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.970212    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-tmp-dir" pod "fc6e0560-7627-4f0d-981e-e607a2d5598d" (UID: "fc6e0560-7627-4f0d-981e-e607a2d5598d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:54.970538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.970306    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-home" pod "fc6e0560-7627-4f0d-981e-e607a2d5598d" (UID: "fc6e0560-7627-4f0d-981e-e607a2d5598d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:54.970538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.970497    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-model-cache\") pod \"fc6e0560-7627-4f0d-981e-e607a2d5598d\" (UID: \"fc6e0560-7627-4f0d-981e-e607a2d5598d\") "
Oct 29 23:23:54.971066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.971035    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-model-cache" pod "fc6e0560-7627-4f0d-981e-e607a2d5598d" (UID: "fc6e0560-7627-4f0d-981e-e607a2d5598d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:54.971253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.971233    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:54.971253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.971253    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:54.971390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.971261    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:54.972669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.972646    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc6e0560-7627-4f0d-981e-e607a2d5598d-tls-certs" pod "fc6e0560-7627-4f0d-981e-e607a2d5598d" (UID: "fc6e0560-7627-4f0d-981e-e607a2d5598d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:23:54.972817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.972713    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6e0560-7627-4f0d-981e-e607a2d5598d-kube-api-access-cthhc" pod "fc6e0560-7627-4f0d-981e-e607a2d5598d" (UID: "fc6e0560-7627-4f0d-981e-e607a2d5598d"). InnerVolumeSpecName "kube-api-access-cthhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:23:54.972817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:54.972757    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-dshm" pod "fc6e0560-7627-4f0d-981e-e607a2d5598d" (UID: "fc6e0560-7627-4f0d-981e-e607a2d5598d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:55.072631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.072595    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e0560-7627-4f0d-981e-e607a2d5598d-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:55.072631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.072627    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e0560-7627-4f0d-981e-e607a2d5598d-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:55.072631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.072639    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cthhc\" (UniqueName: \"kubernetes.io/projected/fc6e0560-7627-4f0d-981e-e607a2d5598d-kube-api-access-cthhc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:23:55.108572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.108493    4827 generic.go:356] "Generic (PLEG): container finished" podID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerID="7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b" exitCode=0
Oct 29 23:23:55.108572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.108532    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"
Oct 29 23:23:55.108760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.108574    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" event={"ID":"fc6e0560-7627-4f0d-981e-e607a2d5598d","Type":"ContainerDied","Data":"7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b"}
Oct 29 23:23:55.108760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.108616    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x" event={"ID":"fc6e0560-7627-4f0d-981e-e607a2d5598d","Type":"ContainerDied","Data":"0034748cc25b7b62c6c8d681a8817ef854e41b245d8c3a41e231d7bf728f73d4"}
Oct 29 23:23:55.108760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.108634    4827 scope.go:122] "RemoveContainer" containerID="7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b"
Oct 29 23:23:55.120246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.120227    4827 scope.go:122] "RemoveContainer" containerID="7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b"
Oct 29 23:23:55.120511 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:23:55.120494    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b\": container with ID starting with 7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b not found: ID does not exist" containerID="7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b"
Oct 29 23:23:55.120567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.120519    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b"} err="failed to get container status \"7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b\": rpc error: code = NotFound desc = could not find container \"7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b\": container with ID starting with 7ecc7fec30e1bc089b1b981193c81f11890ef19dbb1a7a57327ca99811ad671b not found: ID does not exist"
Oct 29 23:23:55.161136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.161098    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"]
Oct 29 23:23:55.165471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:55.165449    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9e7cc9/test-model-9e7cc9-kserve-69798c4b9f-4vc8x"]
Oct 29 23:23:56.491656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:56.491616    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" path="/var/lib/kubelet/pods/fc6e0560-7627-4f0d-981e-e607a2d5598d/volumes"
Oct 29 23:23:59.577721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:23:59.577530    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"]
Oct 29 23:24:02.828398 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:02.828358    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:24:04.281004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:04.280970    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" podUID="2063e478-02e9-4920-a461-50223a414968" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:24:09.539443 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.539376    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" containerID="cri-o://364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe" gracePeriod=45
Oct 29 23:24:09.626493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.626450    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main" probeResult="failure" output="Get \"https://10.134.1.33:8000/ready\": dial tcp 10.134.1.33:8000: connect: connection refused"
Oct 29 23:24:09.849620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.849589    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:24:09.935800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.933981    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c9dd68-a81b-44ed-9430-b0168a557e39-tls-certs\" (UniqueName: \"kubernetes.io/secret/15c9dd68-a81b-44ed-9430-b0168a557e39-tls-certs\") pod \"15c9dd68-a81b-44ed-9430-b0168a557e39\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") "
Oct 29 23:24:09.935800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.934161    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c9dd68-a81b-44ed-9430-b0168a557e39-kube-api-access-djk2s\" (UniqueName: \"kubernetes.io/projected/15c9dd68-a81b-44ed-9430-b0168a557e39-kube-api-access-djk2s\") pod \"15c9dd68-a81b-44ed-9430-b0168a557e39\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") "
Oct 29 23:24:09.935800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.934354    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-dshm\") pod \"15c9dd68-a81b-44ed-9430-b0168a557e39\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") "
Oct 29 23:24:09.935800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.934428    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-tmp-dir\") pod \"15c9dd68-a81b-44ed-9430-b0168a557e39\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") "
Oct 29 23:24:09.935800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.934548    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-home\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-home\") pod \"15c9dd68-a81b-44ed-9430-b0168a557e39\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") "
Oct 29 23:24:09.935800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.934607    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-model-cache\") pod \"15c9dd68-a81b-44ed-9430-b0168a557e39\" (UID: \"15c9dd68-a81b-44ed-9430-b0168a557e39\") "
Oct 29 23:24:09.935800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.935666    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-model-cache" pod "15c9dd68-a81b-44ed-9430-b0168a557e39" (UID: "15c9dd68-a81b-44ed-9430-b0168a557e39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:09.948796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.944032    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-tmp-dir" pod "15c9dd68-a81b-44ed-9430-b0168a557e39" (UID: "15c9dd68-a81b-44ed-9430-b0168a557e39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:09.948796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.944236    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-home" pod "15c9dd68-a81b-44ed-9430-b0168a557e39" (UID: "15c9dd68-a81b-44ed-9430-b0168a557e39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:09.965016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.962979    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c9dd68-a81b-44ed-9430-b0168a557e39-tls-certs" pod "15c9dd68-a81b-44ed-9430-b0168a557e39" (UID: "15c9dd68-a81b-44ed-9430-b0168a557e39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:24:09.970406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.970320    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-dshm" pod "15c9dd68-a81b-44ed-9430-b0168a557e39" (UID: "15c9dd68-a81b-44ed-9430-b0168a557e39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:09.972605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:09.972561    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c9dd68-a81b-44ed-9430-b0168a557e39-kube-api-access-djk2s" pod "15c9dd68-a81b-44ed-9430-b0168a557e39" (UID: "15c9dd68-a81b-44ed-9430-b0168a557e39"). InnerVolumeSpecName "kube-api-access-djk2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:10.068919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.037451    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:10.068919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.037481    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:10.068919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.037494    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:10.068919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.037506    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c9dd68-a81b-44ed-9430-b0168a557e39-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:10.068919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.037517    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c9dd68-a81b-44ed-9430-b0168a557e39-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:10.068919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.037530    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djk2s\" (UniqueName: \"kubernetes.io/projected/15c9dd68-a81b-44ed-9430-b0168a557e39-kube-api-access-djk2s\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:10.317895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.317175    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"
Oct 29 23:24:10.317895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.317099    4827 generic.go:356] "Generic (PLEG): container finished" podID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerID="364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe" exitCode=0
Oct 29 23:24:10.317895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.317518    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" event={"ID":"15c9dd68-a81b-44ed-9430-b0168a557e39","Type":"ContainerDied","Data":"364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe"}
Oct 29 23:24:10.317895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.317561    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf" event={"ID":"15c9dd68-a81b-44ed-9430-b0168a557e39","Type":"ContainerDied","Data":"a9720cc0abe8952a85355af07441bb214156448c4a847496b564275f1457c9c1"}
Oct 29 23:24:10.317895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.317584    4827 scope.go:122] "RemoveContainer" containerID="364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe"
Oct 29 23:24:10.352425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.352403    4827 scope.go:122] "RemoveContainer" containerID="364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe"
Oct 29 23:24:10.352746 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:10.352719    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe\": container with ID starting with 364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe not found: ID does not exist" containerID="364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe"
Oct 29 23:24:10.352854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.352778    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe"} err="failed to get container status \"364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe\": rpc error: code = NotFound desc = could not find container \"364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe\": container with ID starting with 364614de7ffd292bc7d7ee3d2a923710a0a66a5e34478f53bc78e4189ef5f6fe not found: ID does not exist"
Oct 29 23:24:10.380989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.380956    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"]
Oct 29 23:24:10.388350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.388320    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1056cc/test-model-1056cc-kserve-7f6b498f95-gm4mf"]
Oct 29 23:24:10.419287 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:10.419247    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c9dd68_a81b_44ed_9430_b0168a557e39.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c9dd68_a81b_44ed_9430_b0168a557e39.slice/crio-a9720cc0abe8952a85355af07441bb214156448c4a847496b564275f1457c9c1\": RecentStats: unable to find data in memory cache]"
Oct 29 23:24:10.602851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.598442    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" path="/var/lib/kubelet/pods/15c9dd68-a81b-44ed-9430-b0168a557e39/volumes"
Oct 29 23:24:10.680789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:10.680004    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"]
Oct 29 23:24:11.666949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:11.661516    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:24:12.151955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.151931    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:24:12.314837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.307505    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2063e478-02e9-4920-a461-50223a414968-kube-api-access-sl6jc\" (UniqueName: \"kubernetes.io/projected/2063e478-02e9-4920-a461-50223a414968-kube-api-access-sl6jc\") pod \"2063e478-02e9-4920-a461-50223a414968\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") "
Oct 29 23:24:12.314837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.307792    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2063e478-02e9-4920-a461-50223a414968-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063e478-02e9-4920-a461-50223a414968-plugins-config-volume\") pod \"2063e478-02e9-4920-a461-50223a414968\" (UID: \"2063e478-02e9-4920-a461-50223a414968\") "
Oct 29 23:24:12.324363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.324308    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2063e478-02e9-4920-a461-50223a414968-kube-api-access-sl6jc" pod "2063e478-02e9-4920-a461-50223a414968" (UID: "2063e478-02e9-4920-a461-50223a414968"). InnerVolumeSpecName "kube-api-access-sl6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:12.331559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.331519    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2063e478-02e9-4920-a461-50223a414968-plugins-config-volume" pod "2063e478-02e9-4920-a461-50223a414968" (UID: "2063e478-02e9-4920-a461-50223a414968"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:12.410366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.410335    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl6jc\" (UniqueName: \"kubernetes.io/projected/2063e478-02e9-4920-a461-50223a414968-kube-api-access-sl6jc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:12.410366 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.410363    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063e478-02e9-4920-a461-50223a414968-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:12.456077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.456044    4827 generic.go:356] "Generic (PLEG): container finished" podID="2063e478-02e9-4920-a461-50223a414968" containerID="43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b" exitCode=0
Oct 29 23:24:12.456260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.456090    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"
Oct 29 23:24:12.456260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.456125    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" event={"ID":"2063e478-02e9-4920-a461-50223a414968","Type":"ContainerDied","Data":"43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b"}
Oct 29 23:24:12.456260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.456178    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn" event={"ID":"2063e478-02e9-4920-a461-50223a414968","Type":"ContainerDied","Data":"aaf246d62830cbeb5f311d116dd3e0f15ee64f99f788f14b09c78a746416e567"}
Oct 29 23:24:12.456260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.456203    4827 scope.go:122] "RemoveContainer" containerID="43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b"
Oct 29 23:24:12.468200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.468183    4827 scope.go:122] "RemoveContainer" containerID="43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b"
Oct 29 23:24:12.468492 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:12.468471    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b\": container with ID starting with 43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b not found: ID does not exist" containerID="43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b"
Oct 29 23:24:12.468571 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.468502    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b"} err="failed to get container status \"43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b\": rpc error: code = NotFound desc = could not find container \"43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b\": container with ID starting with 43e72f318543e44a7ecd7ad51fa351df47f7b68f6a48f3d47bdcff3a1c3b0f7b not found: ID does not exist"
Oct 29 23:24:12.508803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.508730    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"]
Oct 29 23:24:12.519161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:12.519135    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-462fb2-77cfcbcc5f-z8gvn"]
Oct 29 23:24:14.493122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:14.493090    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2063e478-02e9-4920-a461-50223a414968" path="/var/lib/kubelet/pods/2063e478-02e9-4920-a461-50223a414968/volumes"
Oct 29 23:24:14.601541 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:14.601503    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main" containerID="cri-o://97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb" gracePeriod=45
Oct 29 23:24:14.643834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:14.643799    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": dial tcp 10.134.1.37:8000: connect: connection refused"
Oct 29 23:24:14.905327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:14.905299    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:24:15.055537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055452    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-home\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-home\") pod \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") "
Oct 29 23:24:15.055537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055505    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tmp-dir\") pod \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") "
Oct 29 23:24:15.055759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055614    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tls-certs\") pod \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") "
Oct 29 23:24:15.055759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055740    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-home" pod "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" (UID: "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:15.055859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055805    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tmp-dir" pod "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" (UID: "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:15.055859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055846    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-kube-api-access-hftt8\" (UniqueName: \"kubernetes.io/projected/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-kube-api-access-hftt8\") pod \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") "
Oct 29 23:24:15.055925 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055913    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-dshm\") pod \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") "
Oct 29 23:24:15.055972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.055955    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-model-cache\") pod \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\" (UID: \"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382\") "
Oct 29 23:24:15.056277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.056247    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-model-cache" pod "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" (UID: "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:15.056955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.056933    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:15.057058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.056961    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:15.057058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.056977    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:15.058149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.058122    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-dshm" pod "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" (UID: "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:15.058149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.058136    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-kube-api-access-hftt8" pod "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" (UID: "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382"). InnerVolumeSpecName "kube-api-access-hftt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:15.058280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.058259    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tls-certs" pod "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" (UID: "0e3c9443-ac68-4eb5-aeb0-e5a33b38a382"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:24:15.158394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.158354    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hftt8\" (UniqueName: \"kubernetes.io/projected/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-kube-api-access-hftt8\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:15.158394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.158383    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:15.158394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.158397    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:15.554812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.545857    4827 generic.go:356] "Generic (PLEG): container finished" podID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerID="97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb" exitCode=0
Oct 29 23:24:15.554812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.545973    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" event={"ID":"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382","Type":"ContainerDied","Data":"97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb"}
Oct 29 23:24:15.554812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.546009    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d" event={"ID":"0e3c9443-ac68-4eb5-aeb0-e5a33b38a382","Type":"ContainerDied","Data":"423cb02a46cda5c391215b220b8007ec409c1d9691ad2892d0d92639499cadce"}
Oct 29 23:24:15.554812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.546035    4827 scope.go:122] "RemoveContainer" containerID="97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb"
Oct 29 23:24:15.554812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.546200    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"
Oct 29 23:24:15.582009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.581880    4827 scope.go:122] "RemoveContainer" containerID="97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb"
Oct 29 23:24:15.582226 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:15.582202    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb\": container with ID starting with 97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb not found: ID does not exist" containerID="97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb"
Oct 29 23:24:15.582278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.582235    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb"} err="failed to get container status \"97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb\": rpc error: code = NotFound desc = could not find container \"97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb\": container with ID starting with 97444b21e8d323bcb9293f3f31024c492fe70ee87688f37a468ad5708d22a8fb not found: ID does not exist"
Oct 29 23:24:15.615566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.615542    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"]
Oct 29 23:24:15.625296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.625275    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a9ae14/test-model-a9ae14-kserve-7d9bcddbdf-t2g6d"]
Oct 29 23:24:15.681190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:15.681152    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" podUID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" containerName="payload-pre-processing" containerID="cri-o://7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941" gracePeriod=25
Oct 29 23:24:16.490350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:16.490316    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" path="/var/lib/kubelet/pods/0e3c9443-ac68-4eb5-aeb0-e5a33b38a382/volumes"
Oct 29 23:24:21.556295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:21.556257    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"]
Oct 29 23:24:21.556868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:21.556549    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd" podUID="a94dd682-8a3b-4067-9414-3b6041b34051" containerName="maas-api" containerID="cri-o://027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8" gracePeriod=30
Oct 29 23:24:22.021179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.021136    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" podUID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:24:22.396555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.396521    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:24:22.572455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.572419    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-metrics-certs\") pod \"a94dd682-8a3b-4067-9414-3b6041b34051\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") "
Oct 29 23:24:22.573172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.573145    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94dd682-8a3b-4067-9414-3b6041b34051-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94dd682-8a3b-4067-9414-3b6041b34051-openshift-service-ca\") pod \"a94dd682-8a3b-4067-9414-3b6041b34051\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") "
Oct 29 23:24:22.573488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.573470    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94dd682-8a3b-4067-9414-3b6041b34051-kube-api-access-6cxgt\" (UniqueName: \"kubernetes.io/projected/a94dd682-8a3b-4067-9414-3b6041b34051-kube-api-access-6cxgt\") pod \"a94dd682-8a3b-4067-9414-3b6041b34051\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") "
Oct 29 23:24:22.573783 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.573748    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls\") pod \"a94dd682-8a3b-4067-9414-3b6041b34051\" (UID: \"a94dd682-8a3b-4067-9414-3b6041b34051\") "
Oct 29 23:24:22.576589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.576560    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94dd682-8a3b-4067-9414-3b6041b34051-openshift-service-ca" pod "a94dd682-8a3b-4067-9414-3b6041b34051" (UID: "a94dd682-8a3b-4067-9414-3b6041b34051"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:22.595337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.594804    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls" pod "a94dd682-8a3b-4067-9414-3b6041b34051" (UID: "a94dd682-8a3b-4067-9414-3b6041b34051"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:24:22.595337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.594964    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94dd682-8a3b-4067-9414-3b6041b34051-kube-api-access-6cxgt" pod "a94dd682-8a3b-4067-9414-3b6041b34051" (UID: "a94dd682-8a3b-4067-9414-3b6041b34051"). InnerVolumeSpecName "kube-api-access-6cxgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:22.595704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.595676    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-metrics-certs" pod "a94dd682-8a3b-4067-9414-3b6041b34051" (UID: "a94dd682-8a3b-4067-9414-3b6041b34051"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:24:22.607163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.607133    4827 generic.go:356] "Generic (PLEG): container finished" podID="a94dd682-8a3b-4067-9414-3b6041b34051" containerID="027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8" exitCode=0
Oct 29 23:24:22.607313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.607197    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"
Oct 29 23:24:22.607313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.607254    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd" event={"ID":"a94dd682-8a3b-4067-9414-3b6041b34051","Type":"ContainerDied","Data":"027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8"}
Oct 29 23:24:22.607411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.607313    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd" event={"ID":"a94dd682-8a3b-4067-9414-3b6041b34051","Type":"ContainerDied","Data":"56faad88deb2ded253fb39e9ee76abc1c859ea10f4293a12efc5ced33cc26098"}
Oct 29 23:24:22.607411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.607341    4827 scope.go:122] "RemoveContainer" containerID="027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8"
Oct 29 23:24:22.641861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.641835    4827 scope.go:122] "RemoveContainer" containerID="027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8"
Oct 29 23:24:22.642247 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:22.642221    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8\": container with ID starting with 027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8 not found: ID does not exist" containerID="027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8"
Oct 29 23:24:22.642321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.642262    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8"} err="failed to get container status \"027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8\": rpc error: code = NotFound desc = could not find container \"027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8\": container with ID starting with 027cbbcd9a877d9da56a297c89ccf31a26f7aa3b4ffcf92b18381959cc8d96e8 not found: ID does not exist"
Oct 29 23:24:22.675837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.675739    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"]
Oct 29 23:24:22.677224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.677110    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxgt\" (UniqueName: \"kubernetes.io/projected/a94dd682-8a3b-4067-9414-3b6041b34051-kube-api-access-6cxgt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:22.677224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.677143    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:22.677224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.677157    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94dd682-8a3b-4067-9414-3b6041b34051-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:22.677224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.677169    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94dd682-8a3b-4067-9414-3b6041b34051-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:22.681252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:22.681228    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-462fb2-58ddc48586-ztdqd"]
Oct 29 23:24:24.490440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:24.490411    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94dd682-8a3b-4067-9414-3b6041b34051" path="/var/lib/kubelet/pods/a94dd682-8a3b-4067-9414-3b6041b34051/volumes"
Oct 29 23:24:30.253580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:30.253545    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"]
Oct 29 23:24:30.254126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:30.253822    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" containerID="cri-o://a9d899a8d728de2b8b0de7d56c28cf14058e19401a786ad163ec8abce6af5a05" gracePeriod=30
Oct 29 23:24:31.417493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:31.417456    4827 patch_prober.go:29] interesting pod/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:24:31.417903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:31.417526    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:24:31.835063 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:31.834922    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" podUID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:24:32.825715 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:32.825677    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:24:35.772622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:35.772590    4827 generic.go:356] "Generic (PLEG): container finished" podID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerID="a9d899a8d728de2b8b0de7d56c28cf14058e19401a786ad163ec8abce6af5a05" exitCode=0
Oct 29 23:24:35.773097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:35.772665    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" event={"ID":"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1","Type":"ContainerDied","Data":"a9d899a8d728de2b8b0de7d56c28cf14058e19401a786ad163ec8abce6af5a05"}
Oct 29 23:24:35.773097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:35.772731    4827 scope.go:122] "RemoveContainer" containerID="27e67eb0e647f75e3d148330e20a1bb6e3253dccb8ce6d11ace6e36b53f0eec5"
Oct 29 23:24:35.980046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:35.980017    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:24:36.033546 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.033521    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:24:36.082242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082208    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-envoy\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.082475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082280    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-socket\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.082475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082346    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-plugins-config-volume\") pod \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") "
Oct 29 23:24:36.082475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082427    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istiod-ca-cert\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.082475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082470    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-credential-socket\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.082723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082564    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-kube-api-access-5745r\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-kube-api-access-5745r\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.082723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082590    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-token\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-token\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.082723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082623    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-data\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.082956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082720    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-kube-api-access-974bq\" (UniqueName: \"kubernetes.io/projected/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-kube-api-access-974bq\") pod \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\" (UID: \"233bbfd9-51f0-4b3e-9451-0f391ed9fb75\") "
Oct 29 23:24:36.082956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082753    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-podinfo\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.083080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.082992    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-certs\") pod \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\" (UID: \"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1\") "
Oct 29 23:24:36.084181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.084151    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-certs" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:36.085241 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.085192    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-socket" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:36.085645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.085378    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istiod-ca-cert" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:36.085756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.085641    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-credential-socket" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:36.085994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.085967    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-plugins-config-volume" pod "233bbfd9-51f0-4b3e-9451-0f391ed9fb75" (UID: "233bbfd9-51f0-4b3e-9451-0f391ed9fb75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:36.086623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.086596    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-data" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:36.088177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.088154    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-podinfo" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:24:36.088689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.088670    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-kube-api-access-5745r" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "kube-api-access-5745r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:36.088840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.088751    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-kube-api-access-974bq" pod "233bbfd9-51f0-4b3e-9451-0f391ed9fb75" (UID: "233bbfd9-51f0-4b3e-9451-0f391ed9fb75"). InnerVolumeSpecName "kube-api-access-974bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:36.089025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.088990    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-envoy" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:36.090096 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.090071    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-token" pod "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" (UID: "72ab563d-8ddf-4d47-a003-6aa74a8b9ad1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:36.185529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185494    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185523    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185534    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185543    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185554    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185566    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185580    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5745r\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-kube-api-access-5745r\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185588    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185597    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185605    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974bq\" (UniqueName: \"kubernetes.io/projected/233bbfd9-51f0-4b3e-9451-0f391ed9fb75-kube-api-access-974bq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.185785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.185613    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:36.850789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.844669    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"
Oct 29 23:24:36.865803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.863369    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8" event={"ID":"72ab563d-8ddf-4d47-a003-6aa74a8b9ad1","Type":"ContainerDied","Data":"ead8281b806fbcdc48fa94222aad9da3d241d568060bfd41996e7e55390bcba3"}
Oct 29 23:24:36.865803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.863458    4827 scope.go:122] "RemoveContainer" containerID="a9d899a8d728de2b8b0de7d56c28cf14058e19401a786ad163ec8abce6af5a05"
Oct 29 23:24:36.921665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.918426    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"]
Oct 29 23:24:36.975083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.972069    4827 generic.go:356] "Generic (PLEG): container finished" podID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" containerID="7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941" exitCode=0
Oct 29 23:24:36.975083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.972416    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"
Oct 29 23:24:36.975083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.972446    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" event={"ID":"233bbfd9-51f0-4b3e-9451-0f391ed9fb75","Type":"ContainerDied","Data":"7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941"}
Oct 29 23:24:36.975083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:36.972497    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw" event={"ID":"233bbfd9-51f0-4b3e-9451-0f391ed9fb75","Type":"ContainerDied","Data":"3e302aae788e83a6b8ab24330382e9e1ce26ef7be3a16a652bd4790eed13d8ca"}
Oct 29 23:24:37.026782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.024994    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"]
Oct 29 23:24:37.063819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.063760    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"]
Oct 29 23:24:37.073877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.073834    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e7cc9-56c884b7c5-p7qsw"]
Oct 29 23:24:37.099112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.098804    4827 scope.go:122] "RemoveContainer" containerID="7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941"
Oct 29 23:24:37.209419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.209386    4827 scope.go:122] "RemoveContainer" containerID="7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941"
Oct 29 23:24:37.221712 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:37.221676    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941\": container with ID starting with 7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941 not found: ID does not exist" containerID="7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941"
Oct 29 23:24:37.221987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.221959    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941"} err="failed to get container status \"7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941\": rpc error: code = NotFound desc = could not find container \"7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941\": container with ID starting with 7ee9c399acb693d735f73767b83b829e7b23ed5516f42fe6cf5a16738c87d941 not found: ID does not exist"
Oct 29 23:24:37.232452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.232410    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"]
Oct 29 23:24:37.250384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:37.250288    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-462fb2-openshift-default-58cfd768f5-v2tp8"]
Oct 29 23:24:38.499305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:38.499269    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" path="/var/lib/kubelet/pods/233bbfd9-51f0-4b3e-9451-0f391ed9fb75/volumes"
Oct 29 23:24:38.500109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:38.500089    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" path="/var/lib/kubelet/pods/72ab563d-8ddf-4d47-a003-6aa74a8b9ad1/volumes"
Oct 29 23:24:41.919458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:41.919390    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg" podUID="a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" containerName="payload-processing" containerID="cri-o://4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423" gracePeriod=25
Oct 29 23:24:42.028805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.028729    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" podUID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" containerName="payload-pre-processing" containerID="cri-o://1fa51bdf753d44ad72bbe8d0accae8b578b5e8a9e4f5bf3ef4754a4b7659ba53" gracePeriod=25
Oct 29 23:24:42.551096 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.551065    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:24:42.653827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.646450    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-kube-api-access-xdk7c\" (UniqueName: \"kubernetes.io/projected/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-kube-api-access-xdk7c\") pod \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") "
Oct 29 23:24:42.653827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.646620    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-plugins-config-volume\") pod \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\" (UID: \"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8\") "
Oct 29 23:24:42.653827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.647335    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-plugins-config-volume" pod "a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" (UID: "a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:42.653827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.652554    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-kube-api-access-xdk7c" pod "a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" (UID: "a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8"). InnerVolumeSpecName "kube-api-access-xdk7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:42.749243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.749205    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdk7c\" (UniqueName: \"kubernetes.io/projected/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-kube-api-access-xdk7c\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:42.749243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:42.749235    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.090692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.090655    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" podUID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:24:43.155163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.155124    4827 generic.go:356] "Generic (PLEG): container finished" podID="a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" containerID="4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423" exitCode=0
Oct 29 23:24:43.155332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.155177    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"
Oct 29 23:24:43.155332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.155210    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg" event={"ID":"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8","Type":"ContainerDied","Data":"4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423"}
Oct 29 23:24:43.155332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.155270    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg" event={"ID":"a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8","Type":"ContainerDied","Data":"ab20e216979e28f012112ae2ad25252dfa777c1919bbc888886d0eb31263c4c4"}
Oct 29 23:24:43.155332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.155292    4827 scope.go:122] "RemoveContainer" containerID="4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423"
Oct 29 23:24:43.166350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.166336    4827 scope.go:122] "RemoveContainer" containerID="4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423"
Oct 29 23:24:43.166617 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:43.166592    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423\": container with ID starting with 4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423 not found: ID does not exist" containerID="4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423"
Oct 29 23:24:43.166688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.166624    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423"} err="failed to get container status \"4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423\": rpc error: code = NotFound desc = could not find container \"4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423\": container with ID starting with 4f2ab9ee4aa3a1ce536d78b8f43f2abfd9432fc9cf031d760fd12826fa628423 not found: ID does not exist"
Oct 29 23:24:43.205037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.205010    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"]
Oct 29 23:24:43.214639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:43.214616    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1056cc-7fc98965cb-4lgrg"]
Oct 29 23:24:44.537279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:44.537238    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" path="/var/lib/kubelet/pods/a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8/volumes"
Oct 29 23:24:44.664565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:44.664522    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"]
Oct 29 23:24:44.664813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:44.664786    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" containerID="cri-o://e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060" gracePeriod=30
Oct 29 23:24:46.850542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:46.850500    4827 patch_prober.go:29] interesting pod/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:24:46.850977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:46.850578    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:24:49.959298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:49.959273    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:24:50.109866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.109833    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istiod-ca-cert\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.109910    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-podinfo\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.109987    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-token\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-token\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110083    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-envoy\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110107    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-kube-api-access-kkfp7\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-kube-api-access-kkfp7\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110147    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-socket\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110196    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-credential-socket\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110262    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istiod-ca-cert" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:50.110407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110286    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-data\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110329    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-certs\") pod \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\" (UID: \"9723d28e-b337-4131-a6bd-8b7a6da3ff16\") "
Oct 29 23:24:50.110871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110499    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-socket" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:50.110871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110623    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-credential-socket" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:50.110871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110610    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-data" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:50.110871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.110823    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-certs" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:50.111319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.111289    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.111319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.111314    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.111572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.111329    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.111572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.111343    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.111572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.111356    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.112618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.112594    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-podinfo" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:24:50.113183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.113162    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-kube-api-access-kkfp7" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "kube-api-access-kkfp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:50.113295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.113275    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-envoy" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:50.113538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.113519    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-token" pod "9723d28e-b337-4131-a6bd-8b7a6da3ff16" (UID: "9723d28e-b337-4131-a6bd-8b7a6da3ff16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:50.216334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.215243    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.216334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.215282    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.216334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.215296    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfp7\" (UniqueName: \"kubernetes.io/projected/9723d28e-b337-4131-a6bd-8b7a6da3ff16-kube-api-access-kkfp7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.216334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.215311    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9723d28e-b337-4131-a6bd-8b7a6da3ff16-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:24:50.226570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.226533    4827 generic.go:356] "Generic (PLEG): container finished" podID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerID="e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060" exitCode=0
Oct 29 23:24:50.226743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.226595    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"
Oct 29 23:24:50.226743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.226618    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" event={"ID":"9723d28e-b337-4131-a6bd-8b7a6da3ff16","Type":"ContainerDied","Data":"e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060"}
Oct 29 23:24:50.226743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.226706    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4" event={"ID":"9723d28e-b337-4131-a6bd-8b7a6da3ff16","Type":"ContainerDied","Data":"a553c92c386b497a0129800e14c0e152962dc2aab469d97728d666b16bb20c38"}
Oct 29 23:24:50.226743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.226730    4827 scope.go:122] "RemoveContainer" containerID="e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060"
Oct 29 23:24:50.239428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.239411    4827 scope.go:122] "RemoveContainer" containerID="da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678"
Oct 29 23:24:50.249532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.249511    4827 scope.go:122] "RemoveContainer" containerID="e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060"
Oct 29 23:24:50.249865 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:50.249841    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060\": container with ID starting with e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060 not found: ID does not exist" containerID="e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060"
Oct 29 23:24:50.249974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.249907    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060"} err="failed to get container status \"e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060\": rpc error: code = NotFound desc = could not find container \"e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060\": container with ID starting with e405c63e3f4e3c8a1480b6a1ae986b17f7de56588a8c7ca716e9d546d1be8060 not found: ID does not exist"
Oct 29 23:24:50.249974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.249932    4827 scope.go:122] "RemoveContainer" containerID="da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678"
Oct 29 23:24:50.250392 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:24:50.250365    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678\": container with ID starting with da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678 not found: ID does not exist" containerID="da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678"
Oct 29 23:24:50.250472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.250399    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678"} err="failed to get container status \"da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678\": rpc error: code = NotFound desc = could not find container \"da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678\": container with ID starting with da0538a167dcd1b4fa426c79a9cfefc396cb95c12d326aef7e1724df98386678 not found: ID does not exist"
Oct 29 23:24:50.277071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.277047    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"]
Oct 29 23:24:50.287127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.287104    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9e7cc9-openshift-default-7dcb5c559f-bp7c4"]
Oct 29 23:24:50.490247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:50.490159    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" path="/var/lib/kubelet/pods/9723d28e-b337-4131-a6bd-8b7a6da3ff16/volumes"
Oct 29 23:24:53.091335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:24:53.091300    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" podUID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:25:02.325419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.325180    4827 generic.go:356] "Generic (PLEG): container finished" podID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" containerID="1fa51bdf753d44ad72bbe8d0accae8b578b5e8a9e4f5bf3ef4754a4b7659ba53" exitCode=0
Oct 29 23:25:02.325419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.325248    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" event={"ID":"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4","Type":"ContainerDied","Data":"1fa51bdf753d44ad72bbe8d0accae8b578b5e8a9e4f5bf3ef4754a4b7659ba53"}
Oct 29 23:25:02.325419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.325299    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn" event={"ID":"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4","Type":"ContainerDied","Data":"07938ba368e074f3d938b85ad6a5f71598338acfa88cbb7d9c507a45b8d31119"}
Oct 29 23:25:02.325419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.325323    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07938ba368e074f3d938b85ad6a5f71598338acfa88cbb7d9c507a45b8d31119"
Oct 29 23:25:02.350796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.350755    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:25:02.430422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.430350    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-plugins-config-volume\") pod \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") "
Oct 29 23:25:02.430804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.430756    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-plugins-config-volume" pod "dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" (UID: "dc3921fe-1a6d-4446-9d78-1a6fcaa907b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:02.430865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.430832    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-kube-api-access-54chc\" (UniqueName: \"kubernetes.io/projected/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-kube-api-access-54chc\") pod \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\" (UID: \"dc3921fe-1a6d-4446-9d78-1a6fcaa907b4\") "
Oct 29 23:25:02.431490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.431462    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:02.432907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.432886    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-kube-api-access-54chc" pod "dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" (UID: "dc3921fe-1a6d-4446-9d78-1a6fcaa907b4"). InnerVolumeSpecName "kube-api-access-54chc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:02.511330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.511300    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"]
Oct 29 23:25:02.511593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.511542    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht" podUID="2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" containerName="maas-api" containerID="cri-o://8358e4c89339f098434be212660b2a98c637c730c4ccf058b481d24ff3b30a43" gracePeriod=30
Oct 29 23:25:02.549029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:02.539064    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54chc\" (UniqueName: \"kubernetes.io/projected/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4-kube-api-access-54chc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:03.390551 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:25:03.390508    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:25:03.411423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.411389    4827 generic.go:356] "Generic (PLEG): container finished" podID="2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" containerID="8358e4c89339f098434be212660b2a98c637c730c4ccf058b481d24ff3b30a43" exitCode=0
Oct 29 23:25:03.411681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.411668    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"
Oct 29 23:25:03.412301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.412278    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht" event={"ID":"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e","Type":"ContainerDied","Data":"8358e4c89339f098434be212660b2a98c637c730c4ccf058b481d24ff3b30a43"}
Oct 29 23:25:03.412372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.412360    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht" event={"ID":"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e","Type":"ContainerDied","Data":"327d1cb522640e26388a15187a1d11c33b84de8d5b1d7732e5181ec24b7c5f67"}
Oct 29 23:25:03.412419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.412383    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="327d1cb522640e26388a15187a1d11c33b84de8d5b1d7732e5181ec24b7c5f67"
Oct 29 23:25:03.416576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.416552    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:25:03.462872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.462796    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"]
Oct 29 23:25:03.473020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.472988    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1056cc-96d87d95-fljhn"]
Oct 29 23:25:03.582712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.582680    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-openshift-service-ca\") pod \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") "
Oct 29 23:25:03.582908 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.582808    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-maas-api-tls\") pod \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") "
Oct 29 23:25:03.583045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.583028    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-kube-api-access-mjf5d\" (UniqueName: \"kubernetes.io/projected/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-kube-api-access-mjf5d\") pod \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") "
Oct 29 23:25:03.583126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.583101    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-openshift-service-ca" pod "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" (UID: "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:03.583291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.583131    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-metrics-certs\") pod \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\" (UID: \"2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e\") "
Oct 29 23:25:03.584148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.584122    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:03.585537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.585500    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-maas-api-tls" pod "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" (UID: "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:25:03.585645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.585629    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-kube-api-access-mjf5d" pod "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" (UID: "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e"). InnerVolumeSpecName "kube-api-access-mjf5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:03.585870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.585845    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-metrics-certs" pod "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" (UID: "2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:25:03.702085 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.702042    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjf5d\" (UniqueName: \"kubernetes.io/projected/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-kube-api-access-mjf5d\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:03.702427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.702403    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:03.706845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:03.706818    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:04.419225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:04.419189    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"
Oct 29 23:25:04.469125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:04.469098    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"]
Oct 29 23:25:04.479812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:04.479785    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1056cc-64875d5f88-njqht"]
Oct 29 23:25:04.491107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:04.491078    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" path="/var/lib/kubelet/pods/2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e/volumes"
Oct 29 23:25:04.491623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:04.491600    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" path="/var/lib/kubelet/pods/dc3921fe-1a6d-4446-9d78-1a6fcaa907b4/volumes"
Oct 29 23:25:09.555897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.554925    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"]
Oct 29 23:25:09.578574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.578542    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy"
Oct 29 23:25:09.578808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.578789    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy"
Oct 29 23:25:09.578982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.578968    4827 cpu_manager.go:414] "containerMap: removing container" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main"
Oct 29 23:25:09.579076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579063    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main"
Oct 29 23:25:09.579193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579183    4827 cpu_manager.go:414] "containerMap: removing container" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main"
Oct 29 23:25:09.579276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579263    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main"
Oct 29 23:25:09.579362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579352    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2063e478-02e9-4920-a461-50223a414968" containerName="payload-pre-processing"
Oct 29 23:25:09.579440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579429    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2063e478-02e9-4920-a461-50223a414968" containerName="payload-pre-processing"
Oct 29 23:25:09.579520 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579511    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a94dd682-8a3b-4067-9414-3b6041b34051" containerName="maas-api"
Oct 29 23:25:09.579595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579586    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94dd682-8a3b-4067-9414-3b6041b34051" containerName="maas-api"
Oct 29 23:25:09.579720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579704    4827 cpu_manager.go:414] "containerMap: removing container" podUID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" containerName="payload-pre-processing"
Oct 29 23:25:09.579845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579831    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" containerName="payload-pre-processing"
Oct 29 23:25:09.579954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.579943    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main"
Oct 29 23:25:09.580035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580025    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main"
Oct 29 23:25:09.580130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580121    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" containerName="maas-api"
Oct 29 23:25:09.580212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580202    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" containerName="maas-api"
Oct 29 23:25:09.580306 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580298    4827 cpu_manager.go:414] "containerMap: removing container" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy"
Oct 29 23:25:09.580387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580377    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy"
Oct 29 23:25:09.580470 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580460    4827 cpu_manager.go:414] "containerMap: removing container" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy"
Oct 29 23:25:09.580547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580538    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy"
Oct 29 23:25:09.580625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580617    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main"
Oct 29 23:25:09.580696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580687    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main"
Oct 29 23:25:09.580788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580778    4827 cpu_manager.go:414] "containerMap: removing container" podUID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" containerName="payload-pre-processing"
Oct 29 23:25:09.580873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580864    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" containerName="payload-pre-processing"
Oct 29 23:25:09.580975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.580967    4827 cpu_manager.go:414] "containerMap: removing container" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy"
Oct 29 23:25:09.581041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.581033    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy"
Oct 29 23:25:09.581125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.581116    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" containerName="payload-processing"
Oct 29 23:25:09.581195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.581186    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" containerName="payload-processing"
Oct 29 23:25:09.581541 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.581527    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy"
Oct 29 23:25:09.581643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.581635    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="962329a0-0ecc-4b05-a4f5-46ae3ec9a2b5" containerName="main"
Oct 29 23:25:09.581830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.581720    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c9dd68-a81b-44ed-9430-b0168a557e39" containerName="main"
Oct 29 23:25:09.581942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.581929    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2063e478-02e9-4920-a461-50223a414968" containerName="payload-pre-processing"
Oct 29 23:25:09.582015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582007    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94dd682-8a3b-4067-9414-3b6041b34051" containerName="maas-api"
Oct 29 23:25:09.582127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582117    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="233bbfd9-51f0-4b3e-9451-0f391ed9fb75" containerName="payload-pre-processing"
Oct 29 23:25:09.582205 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582197    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3c9443-ac68-4eb5-aeb0-e5a33b38a382" containerName="main"
Oct 29 23:25:09.582283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582275    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7954a9-4f8c-48ee-8c4e-ef3ceed3497e" containerName="maas-api"
Oct 29 23:25:09.582355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582347    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy"
Oct 29 23:25:09.582421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582413    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="772c47d7-cc86-4369-9921-7fbe19d2a3bd" containerName="istio-proxy"
Oct 29 23:25:09.582492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582483    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6e0560-7627-4f0d-981e-e607a2d5598d" containerName="main"
Oct 29 23:25:09.582566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582557    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3921fe-1a6d-4446-9d78-1a6fcaa907b4" containerName="payload-pre-processing"
Oct 29 23:25:09.582647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.582639    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ca3ed2-9f7f-49ec-b1e1-94f93079a2e8" containerName="payload-processing"
Oct 29 23:25:09.602875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.602844    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"]
Oct 29 23:25:09.604118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.603803    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.619277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.619246    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.619423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.619349    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7nm\" (UniqueName: \"kubernetes.io/projected/c34eb078-27e9-4aca-bc74-bbf49c733fd8-kube-api-access-dh7nm\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.619603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.619580    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.628345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.628314    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:25:09.628513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.628498    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:25:09.720876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.720841    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.721042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.720945    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.721110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.721059    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7nm\" (UniqueName: \"kubernetes.io/projected/c34eb078-27e9-4aca-bc74-bbf49c733fd8-kube-api-access-dh7nm\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.721530 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.721502    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.723404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.723384    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.746892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.746867    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7nm\" (UniqueName: \"kubernetes.io/projected/c34eb078-27e9-4aca-bc74-bbf49c733fd8-kube-api-access-dh7nm\") pod \"maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:09.927475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:09.927443    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:10.233300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:10.233272    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"]
Oct 29 23:25:10.235125 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:25:10.235100    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34eb078_27e9_4aca_bc74_bbf49c733fd8.slice/crio-830e8e1d754c82b7a1134c56eaee134ec0b4628bed4a18ec9901cef031a95ef6 WatchSource:0}: Error finding container 830e8e1d754c82b7a1134c56eaee134ec0b4628bed4a18ec9901cef031a95ef6: Status 404 returned error can't find the container with id 830e8e1d754c82b7a1134c56eaee134ec0b4628bed4a18ec9901cef031a95ef6
Oct 29 23:25:10.618219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:10.618182    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w" event={"ID":"c34eb078-27e9-4aca-bc74-bbf49c733fd8","Type":"ContainerStarted","Data":"98d358d6efba0cadca615c4cf31e429dead8feb7a95128e2bd0baf12ead6df3a"}
Oct 29 23:25:10.618638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:10.618241    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w" event={"ID":"c34eb078-27e9-4aca-bc74-bbf49c733fd8","Type":"ContainerStarted","Data":"830e8e1d754c82b7a1134c56eaee134ec0b4628bed4a18ec9901cef031a95ef6"}
Oct 29 23:25:10.661228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:10.661184    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w" podStartSLOduration=1.6611710629999998 podStartE2EDuration="1.661171063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:10.655586084 +0000 UTC m=+2546.278497128" watchObservedRunningTime="2026-10-29 23:25:10.661171063 +0000 UTC m=+2546.284082083"
Oct 29 23:25:11.529135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:11.529101    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"]
Oct 29 23:25:11.529649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:11.529567    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" containerID="cri-o://ed8554e9ba21c641e2efeb405eeff5d089ae329cdf2d8ecd4f9b39d26114871e" gracePeriod=30
Oct 29 23:25:11.637088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:11.636846    4827 generic.go:356] "Generic (PLEG): container finished" podID="c34eb078-27e9-4aca-bc74-bbf49c733fd8" containerID="98d358d6efba0cadca615c4cf31e429dead8feb7a95128e2bd0baf12ead6df3a" exitCode=0
Oct 29 23:25:11.637614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:11.637125    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w" event={"ID":"c34eb078-27e9-4aca-bc74-bbf49c733fd8","Type":"ContainerDied","Data":"98d358d6efba0cadca615c4cf31e429dead8feb7a95128e2bd0baf12ead6df3a"}
Oct 29 23:25:12.813304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.813277    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:12.844572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.844543    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34eb078-27e9-4aca-bc74-bbf49c733fd8-kube-api-access-dh7nm\" (UniqueName: \"kubernetes.io/projected/c34eb078-27e9-4aca-bc74-bbf49c733fd8-kube-api-access-dh7nm\") pod \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") "
Oct 29 23:25:12.844730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.844607    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-xks-ca\") pod \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") "
Oct 29 23:25:12.844850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.844755    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-openshift-ca\") pod \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\" (UID: \"c34eb078-27e9-4aca-bc74-bbf49c733fd8\") "
Oct 29 23:25:12.845161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.845134    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-openshift-ca" pod "c34eb078-27e9-4aca-bc74-bbf49c733fd8" (UID: "c34eb078-27e9-4aca-bc74-bbf49c733fd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:12.845609 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.845587    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:12.846952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.846929    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34eb078-27e9-4aca-bc74-bbf49c733fd8-kube-api-access-dh7nm" pod "c34eb078-27e9-4aca-bc74-bbf49c733fd8" (UID: "c34eb078-27e9-4aca-bc74-bbf49c733fd8"). InnerVolumeSpecName "kube-api-access-dh7nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:12.847041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.846948    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-xks-ca" pod "c34eb078-27e9-4aca-bc74-bbf49c733fd8" (UID: "c34eb078-27e9-4aca-bc74-bbf49c733fd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:25:12.978848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.952372    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh7nm\" (UniqueName: \"kubernetes.io/projected/c34eb078-27e9-4aca-bc74-bbf49c733fd8-kube-api-access-dh7nm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:12.979156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:12.979117    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34eb078-27e9-4aca-bc74-bbf49c733fd8-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:13.204222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:13.204186    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"]
Oct 29 23:25:13.204804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:13.204751    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy" containerID="cri-o://9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312" gracePeriod=30
Oct 29 23:25:13.504686 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:13.504653    4827 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:25:13.504877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:13.504716    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:25:13.656383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:13.656354    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"
Oct 29 23:25:13.656585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:13.656393    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w" event={"ID":"c34eb078-27e9-4aca-bc74-bbf49c733fd8","Type":"ContainerDied","Data":"830e8e1d754c82b7a1134c56eaee134ec0b4628bed4a18ec9901cef031a95ef6"}
Oct 29 23:25:13.656585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:13.656452    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="830e8e1d754c82b7a1134c56eaee134ec0b4628bed4a18ec9901cef031a95ef6"
Oct 29 23:25:16.977606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:16.977118    4827 generic.go:356] "Generic (PLEG): container finished" podID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerID="ed8554e9ba21c641e2efeb405eeff5d089ae329cdf2d8ecd4f9b39d26114871e" exitCode=0
Oct 29 23:25:16.977606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:16.977428    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" event={"ID":"80096ee0-b2d5-4fac-9a73-6a05813c0ff0","Type":"ContainerDied","Data":"ed8554e9ba21c641e2efeb405eeff5d089ae329cdf2d8ecd4f9b39d26114871e"}
Oct 29 23:25:16.977606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:16.977487    4827 scope.go:122] "RemoveContainer" containerID="c694795834d6ca69cb9e0b536662a4ea9eaf4da0c7946af8b3eda05d78163185"
Oct 29 23:25:17.541848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.541821    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:25:17.677542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.677440    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-kube-api-access-n6lzn\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-kube-api-access-n6lzn\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.677542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.677497    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-token\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-token\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.677860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.677621    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-credential-socket\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.677860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.677698    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-podinfo\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.678171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678148    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-data\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.678388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678368    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-socket\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.678616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678587    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istiod-ca-cert\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.678711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678676    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-certs\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.678789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678754    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-envoy\") pod \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\" (UID: \"80096ee0-b2d5-4fac-9a73-6a05813c0ff0\") "
Oct 29 23:25:17.679186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678151    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-credential-socket" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:17.679736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.679704    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-certs" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:17.679736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.679714    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.679953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678366    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-data" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:17.679953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.678540    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-socket" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:17.679953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.679163    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istiod-ca-cert" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:17.681112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.681073    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-kube-api-access-n6lzn" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "kube-api-access-n6lzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:17.681621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.681593    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-envoy" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:17.681952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.681923    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-token" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:17.682580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.682554    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-podinfo" pod "80096ee0-b2d5-4fac-9a73-6a05813c0ff0" (UID: "80096ee0-b2d5-4fac-9a73-6a05813c0ff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:25:17.781384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781336    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.781384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781380    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6lzn\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-kube-api-access-n6lzn\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.781384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781393    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.781640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781405    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.781640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781416    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.781640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781427    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.781640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781438    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.781640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.781449    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80096ee0-b2d5-4fac-9a73-6a05813c0ff0-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:17.986491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.986421    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"
Oct 29 23:25:17.986491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.986468    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk" event={"ID":"80096ee0-b2d5-4fac-9a73-6a05813c0ff0","Type":"ContainerDied","Data":"db2cb33a01aeed089ad706c6eb4db237e143458fbde0f3104f190a12e4679d4a"}
Oct 29 23:25:17.986997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:17.986525    4827 scope.go:122] "RemoveContainer" containerID="ed8554e9ba21c641e2efeb405eeff5d089ae329cdf2d8ecd4f9b39d26114871e"
Oct 29 23:25:18.036904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.036873    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"]
Oct 29 23:25:18.042949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.042925    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-5nxxk"]
Oct 29 23:25:18.489796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.489707    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" path="/var/lib/kubelet/pods/80096ee0-b2d5-4fac-9a73-6a05813c0ff0/volumes"
Oct 29 23:25:18.498187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.498164    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 23:25:18.595543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.595506    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-podinfo\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.595904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.595881    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-envoy\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.596377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.596347    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-data" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:18.596494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.596390    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-data\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.596494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.596441    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-kube-api-access-hnsgq\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-kube-api-access-hnsgq\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.596832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.596809    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-token\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.597159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.597138    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-certs\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.597383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.597359    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istiod-ca-cert\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.597480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.597436    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-socket\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.597480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.597472    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-credential-socket\") pod \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\" (UID: \"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40\") "
Oct 29 23:25:18.598287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598180    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istiod-ca-cert" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:18.598397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598357    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-podinfo" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:25:18.598618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598467    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-credential-socket" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:18.598709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598651    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-socket" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:18.598792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598708    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-envoy" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:18.598792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598652    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.598792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598779    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.598979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598798    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.598979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.598813    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.599172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.599139    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-certs" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:18.600084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.600060    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-kube-api-access-hnsgq" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "kube-api-access-hnsgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:18.600491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.600464    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-token" pod "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" (UID: "8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:18.720798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.720681    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.720798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.720712    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.720798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.720729    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnsgq\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-kube-api-access-hnsgq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.720798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.720744    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.721549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.720758    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:18.995314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.995281    4827 generic.go:356] "Generic (PLEG): container finished" podID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerID="9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312" exitCode=0
Oct 29 23:25:18.995855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.995329    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"
Oct 29 23:25:18.995855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.995358    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" event={"ID":"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40","Type":"ContainerDied","Data":"9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312"}
Oct 29 23:25:18.995855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.995408    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x" event={"ID":"8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40","Type":"ContainerDied","Data":"16a9bf2b022351050c7927dd83429b81d2f2f0fed89ed46eaf736515fecfed63"}
Oct 29 23:25:18.995855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:18.995434    4827 scope.go:122] "RemoveContainer" containerID="9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312"
Oct 29 23:25:19.006946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:19.006922    4827 scope.go:122] "RemoveContainer" containerID="9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312"
Oct 29 23:25:19.007247 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:25:19.007221    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312\": container with ID starting with 9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312 not found: ID does not exist" containerID="9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312"
Oct 29 23:25:19.007335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:19.007258    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312"} err="failed to get container status \"9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312\": rpc error: code = NotFound desc = could not find container \"9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312\": container with ID starting with 9effda40ed8de9a3bbb4b080dc7cb6a3a680db27ba38e9dc1f2909e85e05e312 not found: ID does not exist"
Oct 29 23:25:19.046133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:19.046107    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"]
Oct 29 23:25:19.056814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:19.056790    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-q9f6x"]
Oct 29 23:25:20.489744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:20.489712    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" path="/var/lib/kubelet/pods/8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40/volumes"
Oct 29 23:25:22.578029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:22.577994    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"]
Oct 29 23:25:26.692882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:26.692804    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"]
Oct 29 23:25:26.693401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:26.693129    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy" containerID="cri-o://3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2" gracePeriod=30
Oct 29 23:25:27.578521 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:27.578483    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t" podUID="5c661a0c-8d4b-4918-a2e1-944661be7cc5" containerName="payload-processing" containerID="cri-o://f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d" gracePeriod=25
Oct 29 23:25:27.888237 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:27.888208    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:25:28.010468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.010437    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c661a0c-8d4b-4918-a2e1-944661be7cc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c661a0c-8d4b-4918-a2e1-944661be7cc5-plugins-config-volume\") pod \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") "
Oct 29 23:25:28.010692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.010670    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c661a0c-8d4b-4918-a2e1-944661be7cc5-kube-api-access-cvsbd\" (UniqueName: \"kubernetes.io/projected/5c661a0c-8d4b-4918-a2e1-944661be7cc5-kube-api-access-cvsbd\") pod \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\" (UID: \"5c661a0c-8d4b-4918-a2e1-944661be7cc5\") "
Oct 29 23:25:28.010848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.010825    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c661a0c-8d4b-4918-a2e1-944661be7cc5-plugins-config-volume" pod "5c661a0c-8d4b-4918-a2e1-944661be7cc5" (UID: "5c661a0c-8d4b-4918-a2e1-944661be7cc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:28.011200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.011184    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c661a0c-8d4b-4918-a2e1-944661be7cc5-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:28.013158 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.013133    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c661a0c-8d4b-4918-a2e1-944661be7cc5-kube-api-access-cvsbd" pod "5c661a0c-8d4b-4918-a2e1-944661be7cc5" (UID: "5c661a0c-8d4b-4918-a2e1-944661be7cc5"). InnerVolumeSpecName "kube-api-access-cvsbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:28.125414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.125380    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvsbd\" (UniqueName: \"kubernetes.io/projected/5c661a0c-8d4b-4918-a2e1-944661be7cc5-kube-api-access-cvsbd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:28.141802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.128290    4827 generic.go:356] "Generic (PLEG): container finished" podID="5c661a0c-8d4b-4918-a2e1-944661be7cc5" containerID="f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d" exitCode=0
Oct 29 23:25:28.141802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.128588    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t" event={"ID":"5c661a0c-8d4b-4918-a2e1-944661be7cc5","Type":"ContainerDied","Data":"f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d"}
Oct 29 23:25:28.141802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.128646    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t" event={"ID":"5c661a0c-8d4b-4918-a2e1-944661be7cc5","Type":"ContainerDied","Data":"148a93e6cc5c187a1fa1a230694d60baa779583e7603eec64796a26ccb611085"}
Oct 29 23:25:28.141802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.128676    4827 scope.go:122] "RemoveContainer" containerID="f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d"
Oct 29 23:25:28.141802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.129068    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"
Oct 29 23:25:28.183900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.183027    4827 scope.go:122] "RemoveContainer" containerID="f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d"
Oct 29 23:25:28.183900 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:25:28.183462    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d\": container with ID starting with f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d not found: ID does not exist" containerID="f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d"
Oct 29 23:25:28.183900 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.183499    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d"} err="failed to get container status \"f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d\": rpc error: code = NotFound desc = could not find container \"f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d\": container with ID starting with f2ef2aa22caa23c1dc3870426d0d212e40add437b7db42d90992d29747b9607d not found: ID does not exist"
Oct 29 23:25:28.203816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.203742    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"]
Oct 29 23:25:28.210479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.210443    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b2b71e-5d7447687b-scv2t"]
Oct 29 23:25:28.490353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:28.490274    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c661a0c-8d4b-4918-a2e1-944661be7cc5" path="/var/lib/kubelet/pods/5c661a0c-8d4b-4918-a2e1-944661be7cc5/volumes"
Oct 29 23:25:32.001207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.001178    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:25:32.085959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.085930    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"]
Oct 29 23:25:32.086805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086780    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5c661a0c-8d4b-4918-a2e1-944661be7cc5" containerName="payload-processing"
Oct 29 23:25:32.086805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086804    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c661a0c-8d4b-4918-a2e1-944661be7cc5" containerName="payload-processing"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086837    4827 cpu_manager.go:414] "containerMap: removing container" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086845    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086877    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c34eb078-27e9-4aca-bc74-bbf49c733fd8" containerName="revoke-keys"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086885    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34eb078-27e9-4aca-bc74-bbf49c733fd8" containerName="revoke-keys"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086895    4827 cpu_manager.go:414] "containerMap: removing container" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086901    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086911    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086918    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086961    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy"
Oct 29 23:25:32.087031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.086968    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087139    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087157    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34eb078-27e9-4aca-bc74-bbf49c733fd8" containerName="revoke-keys"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087166    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerName="istio-proxy"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087175    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9723d28e-b337-4131-a6bd-8b7a6da3ff16" containerName="istio-proxy"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087188    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087199    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea4e64f-9fbe-4d37-a6c0-ffb3f4f71a40" containerName="istio-proxy"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087214    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ab563d-8ddf-4d47-a003-6aa74a8b9ad1" containerName="istio-proxy"
Oct 29 23:25:32.087538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.087224    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c661a0c-8d4b-4918-a2e1-944661be7cc5" containerName="payload-processing"
Oct 29 23:25:32.089337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089310    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istiod-ca-cert\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.089447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089406    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-data\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.089519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089505    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-envoy\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.089564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089540    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-podinfo\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.089621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089596    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-token\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-token\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.089714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089684    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-certs\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.089864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089726    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-data" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:32.089864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089747    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istiod-ca-cert" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:32.090053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089866    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-socket\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.090053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089939    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-kube-api-access-85jb2\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-kube-api-access-85jb2\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.090053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.089977    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-credential-socket\") pod \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\" (UID: \"983a74fa-e694-47c2-8b53-dc3f77d1aedd\") "
Oct 29 23:25:32.090234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.090118    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-certs" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:32.090234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.090220    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-socket" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:32.090833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.090481    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-credential-socket" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:32.090965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.090939    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.091042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.090965    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.091042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.090981    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.091042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.090995    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.091042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.091007    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.091812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.091728    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.093349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.093325    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-podinfo" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:25:32.093449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.093355    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-envoy" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:32.094240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.094216    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-kube-api-access-85jb2" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "kube-api-access-85jb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:32.094538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.094511    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-token" pod "983a74fa-e694-47c2-8b53-dc3f77d1aedd" (UID: "983a74fa-e694-47c2-8b53-dc3f77d1aedd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:32.114309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.107875    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cc6328-openshift-default-dockercfg-2g58g\""
Oct 29 23:25:32.135858 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.135822    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"]
Oct 29 23:25:32.182623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.182584    4827 generic.go:356] "Generic (PLEG): container finished" podID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" containerID="3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2" exitCode=0
Oct 29 23:25:32.182821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.182641    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"
Oct 29 23:25:32.182821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.182663    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" event={"ID":"983a74fa-e694-47c2-8b53-dc3f77d1aedd","Type":"ContainerDied","Data":"3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2"}
Oct 29 23:25:32.182821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.182732    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj" event={"ID":"983a74fa-e694-47c2-8b53-dc3f77d1aedd","Type":"ContainerDied","Data":"ca503c57f0cd0e78cb5c4737ced2907bfd4ebd1cabd516fea1373dc7d5d8a831"}
Oct 29 23:25:32.182821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.182782    4827 scope.go:122] "RemoveContainer" containerID="3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2"
Oct 29 23:25:32.192164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192040    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-podinfo\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.192164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192157    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-token\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.192400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192271    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-credential-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.192400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192301    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-envoy\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192388    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef9cfdd-9924-4e7e-8de1-37510651061a-istiod-ca-cert\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192522    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-data\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192702    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ljt\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-kube-api-access-58ljt\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.192878    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.193251    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-certs\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.193416    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.193434    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.193449    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.193788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.193466    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85jb2\" (UniqueName: \"kubernetes.io/projected/983a74fa-e694-47c2-8b53-dc3f77d1aedd-kube-api-access-85jb2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:32.197989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.197970    4827 scope.go:122] "RemoveContainer" containerID="3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2"
Oct 29 23:25:32.198329 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:25:32.198306    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2\": container with ID starting with 3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2 not found: ID does not exist" containerID="3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2"
Oct 29 23:25:32.198410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.198332    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2"} err="failed to get container status \"3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2\": rpc error: code = NotFound desc = could not find container \"3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2\": container with ID starting with 3818e987612571ea6dd003dce51bb604e79312094f554ff676c02b84442607c2 not found: ID does not exist"
Oct 29 23:25:32.234395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.234329    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"]
Oct 29 23:25:32.238771 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.238738    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-tdrtj"]
Oct 29 23:25:32.294578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.294548    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-certs\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.294711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.294604    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-podinfo\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.294711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.294636    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-token\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.294711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.294683    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-credential-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295000    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-certs\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295030    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-credential-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295081    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-envoy\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295130    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef9cfdd-9924-4e7e-8de1-37510651061a-istiod-ca-cert\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295163    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-data\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295182    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ljt\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-kube-api-access-58ljt\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295308    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295496 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295467    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-data\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295618    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.295890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.295874    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef9cfdd-9924-4e7e-8de1-37510651061a-istiod-ca-cert\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.297165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.297149    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-envoy\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.297647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.297623    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-podinfo\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.316698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.316679    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-token\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.317021 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.317005    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ljt\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-kube-api-access-58ljt\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.475782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.475744    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:32.490677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.490653    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983a74fa-e694-47c2-8b53-dc3f77d1aedd" path="/var/lib/kubelet/pods/983a74fa-e694-47c2-8b53-dc3f77d1aedd/volumes"
Oct 29 23:25:32.804382 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:25:32.804348    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef9cfdd_9924_4e7e_8de1_37510651061a.slice/crio-aa1f27e93b0ca3e1edaa560198a92722937e36d61f5e331a6471b1c859ad06f6 WatchSource:0}: Error finding container aa1f27e93b0ca3e1edaa560198a92722937e36d61f5e331a6471b1c859ad06f6: Status 404 returned error can't find the container with id aa1f27e93b0ca3e1edaa560198a92722937e36d61f5e331a6471b1c859ad06f6
Oct 29 23:25:32.804831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.804793    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"]
Oct 29 23:25:32.807379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.807345    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:25:32.807498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.807415    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:25:32.807498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:32.807446    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:25:32.827551 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:25:32.827522    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:25:33.194227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.194194    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" event={"ID":"fef9cfdd-9924-4e7e-8de1-37510651061a","Type":"ContainerStarted","Data":"31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3"}
Oct 29 23:25:33.194879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.194249    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" event={"ID":"fef9cfdd-9924-4e7e-8de1-37510651061a","Type":"ContainerStarted","Data":"aa1f27e93b0ca3e1edaa560198a92722937e36d61f5e331a6471b1c859ad06f6"}
Oct 29 23:25:33.260622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.260573    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podStartSLOduration=1.260558679 podStartE2EDuration="1.260558679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:33.247913183 +0000 UTC m=+2568.870824227" watchObservedRunningTime="2026-10-29 23:25:33.260558679 +0000 UTC m=+2568.883469700"
Oct 29 23:25:33.346329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.346298    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"]
Oct 29 23:25:33.346583 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.346542    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy" containerID="cri-o://62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9" gracePeriod=30
Oct 29 23:25:33.489837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.482150    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:33.503986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.503949    4827 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 29 23:25:33.504278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:33.504247    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 29 23:25:34.485788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:34.478961    4827 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 29 23:25:34.485788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:34.479029    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 29 23:25:35.480239 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:35.480202    4827 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 29 23:25:35.480535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:35.480506    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 29 23:25:36.488208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:36.488166    4827 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 29 23:25:36.488724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:36.488243    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 29 23:25:37.478819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:37.478300    4827 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 29 23:25:37.478819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:37.478369    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 29 23:25:38.482788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.479273    4827 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 29 23:25:38.482788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.479369    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 29 23:25:38.714790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.705294    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"]
Oct 29 23:25:38.714790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.707285    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy" containerID="cri-o://2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd" gracePeriod=30
Oct 29 23:25:38.760968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.760945    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:25:38.923588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.923556    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istiod-ca-cert\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.923817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.923632    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-token\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-token\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.923817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.923667    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-envoy\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.923817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.923755    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-socket\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.924017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.923860    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-credential-socket\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.924017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.923946    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-podinfo\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.924153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.924131    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-kube-api-access-zgxsq\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-kube-api-access-zgxsq\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.924213 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.924200    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-certs\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.924266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.924253    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-data\") pod \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\" (UID: \"02be85c9-a940-4c56-a4ce-bcd3fffcc72f\") "
Oct 29 23:25:38.925266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.925241    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-data" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:38.925396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.925274    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istiod-ca-cert" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:38.925448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.925402    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-socket" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:38.925448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.925427    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-credential-socket" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:38.925558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.925547    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-certs" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:38.927757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.927733    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-token" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:38.927866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.927780    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-kube-api-access-zgxsq" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "kube-api-access-zgxsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:38.928032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.928014    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-podinfo" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:25:38.928559 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:38.928543    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-envoy" pod "02be85c9-a940-4c56-a4ce-bcd3fffcc72f" (UID: "02be85c9-a940-4c56-a4ce-bcd3fffcc72f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:39.026941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026859    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.026941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026884    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.026941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026893    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.026941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026908    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.026941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026921    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.026941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026935    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.026941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026947    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgxsq\" (UniqueName: \"kubernetes.io/projected/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-kube-api-access-zgxsq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.027326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026957    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.027326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.026968    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02be85c9-a940-4c56-a4ce-bcd3fffcc72f-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:39.297721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.297636    4827 generic.go:356] "Generic (PLEG): container finished" podID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerID="62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9" exitCode=0
Oct 29 23:25:39.297918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.297712    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" event={"ID":"02be85c9-a940-4c56-a4ce-bcd3fffcc72f","Type":"ContainerDied","Data":"62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9"}
Oct 29 23:25:39.297918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.297784    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m" event={"ID":"02be85c9-a940-4c56-a4ce-bcd3fffcc72f","Type":"ContainerDied","Data":"0ce67dd5baa51a87197e126a7a3590dad0692edab42058da1e593a0f21b05a2f"}
Oct 29 23:25:39.297918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.297816    4827 scope.go:122] "RemoveContainer" containerID="62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9"
Oct 29 23:25:39.298170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.298152    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"
Oct 29 23:25:39.310665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.310642    4827 scope.go:122] "RemoveContainer" containerID="62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9"
Oct 29 23:25:39.310994 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:25:39.310975    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9\": container with ID starting with 62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9 not found: ID does not exist" containerID="62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9"
Oct 29 23:25:39.311074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.311005    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9"} err="failed to get container status \"62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9\": rpc error: code = NotFound desc = could not find container \"62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9\": container with ID starting with 62872d82ddedba578e4951bb000b6f9affc587bf4815aaa01fb87c3890475ec9 not found: ID does not exist"
Oct 29 23:25:39.358407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.358382    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"]
Oct 29 23:25:39.394819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.394781    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bn64m"]
Oct 29 23:25:39.478185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.478151    4827 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 29 23:25:39.478471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:39.478444    4827 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 29 23:25:40.490725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:40.490689    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" path="/var/lib/kubelet/pods/02be85c9-a940-4c56-a4ce-bcd3fffcc72f/volumes"
Oct 29 23:25:40.491138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:40.491123    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:40.491413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:40.491395    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:40.492099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:40.492079    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:25:44.001648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.001619    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 23:25:44.048989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.048960    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istiod-ca-cert\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049006    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-socket\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049187 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049041    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-envoy\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049189    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-data\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049231    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-kube-api-access-gjftw\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-kube-api-access-gjftw\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049302    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-socket" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:44.049426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049354    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-token\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-token\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049406    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-credential-socket\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049452    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-certs\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049507    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-podinfo\") pod \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\" (UID: \"9f572783-4d2e-47bd-a5e2-543d7d1526a5\") "
Oct 29 23:25:44.049596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049540    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-data" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:44.049596 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.049447    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istiod-ca-cert" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:44.050195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.050164    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-credential-socket" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:44.050303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.050266    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.050303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.050288    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.050303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.050301    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.050660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.050572    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-certs" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:44.052453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.052417    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-podinfo" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:25:44.058152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.058089    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-envoy" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:25:44.059713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.058173    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-token" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:44.060448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.060425    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-kube-api-access-gjftw" pod "9f572783-4d2e-47bd-a5e2-543d7d1526a5" (UID: "9f572783-4d2e-47bd-a5e2-543d7d1526a5"). InnerVolumeSpecName "kube-api-access-gjftw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:44.152005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.151967    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.152005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.151998    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjftw\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-kube-api-access-gjftw\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.152005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.152009    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.152232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.152017    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.152232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.152027    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f572783-4d2e-47bd-a5e2-543d7d1526a5-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.152232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.152035    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f572783-4d2e-47bd-a5e2-543d7d1526a5-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:25:44.342419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.342333    4827 generic.go:356] "Generic (PLEG): container finished" podID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerID="2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd" exitCode=0
Oct 29 23:25:44.342419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.342384    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"
Oct 29 23:25:44.342683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.342416    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" event={"ID":"9f572783-4d2e-47bd-a5e2-543d7d1526a5","Type":"ContainerDied","Data":"2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd"}
Oct 29 23:25:44.342683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.342480    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb" event={"ID":"9f572783-4d2e-47bd-a5e2-543d7d1526a5","Type":"ContainerDied","Data":"bb19c5c4451536ae2be99090a03648643293288a65b92179d16b2eb8311698f6"}
Oct 29 23:25:44.342683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.342511    4827 scope.go:122] "RemoveContainer" containerID="2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd"
Oct 29 23:25:44.354244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.354226    4827 scope.go:122] "RemoveContainer" containerID="2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd"
Oct 29 23:25:44.354527 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:25:44.354506    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd\": container with ID starting with 2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd not found: ID does not exist" containerID="2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd"
Oct 29 23:25:44.354617 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.354538    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd"} err="failed to get container status \"2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd\": rpc error: code = NotFound desc = could not find container \"2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd\": container with ID starting with 2e138539f0995bdd144aca90612387da071fc800ec10ca5a6b05bcb1e52abafd not found: ID does not exist"
Oct 29 23:25:44.393808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.393759    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"]
Oct 29 23:25:44.402290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.402266    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-ssdgb"]
Oct 29 23:25:44.501087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:44.501057    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" path="/var/lib/kubelet/pods/9f572783-4d2e-47bd-a5e2-543d7d1526a5/volumes"
Oct 29 23:25:50.616415 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:50.616381    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"]
Oct 29 23:25:50.626188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:50.626156    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-faa4fc-6nv8w"]
Oct 29 23:25:51.058453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:51.058422    4827 scope.go:122] "RemoveContainer" containerID="1fa51bdf753d44ad72bbe8d0accae8b578b5e8a9e4f5bf3ef4754a4b7659ba53"
Oct 29 23:25:51.069740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:51.069721    4827 scope.go:122] "RemoveContainer" containerID="8358e4c89339f098434be212660b2a98c637c730c4ccf058b481d24ff3b30a43"
Oct 29 23:25:51.861875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:51.861839    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"]
Oct 29 23:25:52.490731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:52.490691    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34eb078-27e9-4aca-bc74-bbf49c733fd8" path="/var/lib/kubelet/pods/c34eb078-27e9-4aca-bc74-bbf49c733fd8/volumes"
Oct 29 23:25:58.495814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:58.490964    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"]
Oct 29 23:25:58.495814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:25:58.491192    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" containerID="cri-o://696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc" gracePeriod=30
Oct 29 23:26:02.184325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:02.184286    4827 patch_prober.go:29] interesting pod/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:26:02.184709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:02.184352    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:26:02.824929 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:02.824888    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:26:03.890374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.890349    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:26:03.946472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.946436    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-envoy\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.946708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.946533    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547cb68c-1fcb-41c4-944f-246ffafd7f21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547cb68c-1fcb-41c4-944f-246ffafd7f21-istiod-ca-cert\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.946708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.946585    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-podinfo\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.946903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.946760    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-socket\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.946903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.946848    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-credential-socket\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.947020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.946913    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-token\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-token\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.947319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.947296    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-socket" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:03.947605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.947579    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547cb68c-1fcb-41c4-944f-246ffafd7f21-istiod-ca-cert" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:26:03.948423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.947844    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-credential-socket" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:03.948423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.948164    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-certs\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.948423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.948220    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-kube-api-access-7k8rd\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-kube-api-access-7k8rd\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.948423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.948266    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-data\") pod \"547cb68c-1fcb-41c4-944f-246ffafd7f21\" (UID: \"547cb68c-1fcb-41c4-944f-246ffafd7f21\") "
Oct 29 23:26:03.951136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.948964    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:03.951136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.948983    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547cb68c-1fcb-41c4-944f-246ffafd7f21-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:03.951136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.948998    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:03.951136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.949268    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-certs" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:03.951136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.949516    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-data" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:03.951136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.950036    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-podinfo" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:26:03.954111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.954071    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-envoy" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:03.954287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.954187    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-kube-api-access-7k8rd" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "kube-api-access-7k8rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:03.954490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:03.954458    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-token" pod "547cb68c-1fcb-41c4-944f-246ffafd7f21" (UID: "547cb68c-1fcb-41c4-944f-246ffafd7f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:04.050283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.050195    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:04.050283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.050223    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:04.050283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.050233    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8rd\" (UniqueName: \"kubernetes.io/projected/547cb68c-1fcb-41c4-944f-246ffafd7f21-kube-api-access-7k8rd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:04.050283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.050242    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:04.050283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.050250    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:04.050283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.050258    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547cb68c-1fcb-41c4-944f-246ffafd7f21-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:04.572204 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.572174    4827 generic.go:356] "Generic (PLEG): container finished" podID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerID="696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc" exitCode=0
Oct 29 23:26:04.572458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.572223    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"
Oct 29 23:26:04.572458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.572260    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" event={"ID":"547cb68c-1fcb-41c4-944f-246ffafd7f21","Type":"ContainerDied","Data":"696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc"}
Oct 29 23:26:04.572458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.572325    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s" event={"ID":"547cb68c-1fcb-41c4-944f-246ffafd7f21","Type":"ContainerDied","Data":"3fb9aefb0272190a759dbb9fb8462212b2ede3c30b1369ebfb35831dda468d7e"}
Oct 29 23:26:04.572458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.572354    4827 scope.go:122] "RemoveContainer" containerID="696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc"
Oct 29 23:26:04.583396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.583380    4827 scope.go:122] "RemoveContainer" containerID="063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f"
Oct 29 23:26:04.593793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.593759    4827 scope.go:122] "RemoveContainer" containerID="696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc"
Oct 29 23:26:04.594078 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:04.594059    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc\": container with ID starting with 696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc not found: ID does not exist" containerID="696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc"
Oct 29 23:26:04.594139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.594088    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc"} err="failed to get container status \"696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc\": rpc error: code = NotFound desc = could not find container \"696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc\": container with ID starting with 696b0655715dac4109b48b9c2c97b79f9873264c2bdcaf1e88f290f42e40f9fc not found: ID does not exist"
Oct 29 23:26:04.594139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.594109    4827 scope.go:122] "RemoveContainer" containerID="063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f"
Oct 29 23:26:04.594338 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:04.594322    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f\": container with ID starting with 063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f not found: ID does not exist" containerID="063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f"
Oct 29 23:26:04.594391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.594342    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f"} err="failed to get container status \"063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f\": rpc error: code = NotFound desc = could not find container \"063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f\": container with ID starting with 063ffbcee9c9c12caef1f1940de3927ca369f1603e2c0d7f54f5daead1de598f not found: ID does not exist"
Oct 29 23:26:04.632899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.632873    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"]
Oct 29 23:26:04.643654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:04.643537    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-faa4fc-openshift-default-69cb8b4565-mdj5s"]
Oct 29 23:26:06.492755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:06.492722    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" path="/var/lib/kubelet/pods/547cb68c-1fcb-41c4-944f-246ffafd7f21/volumes"
Oct 29 23:26:06.891003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:06.890959    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main" containerID="cri-o://97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d" gracePeriod=45
Oct 29 23:26:07.185409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.185381    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:26:07.214907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.214307    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-dshm\") pod \"afd19809-a784-4509-bdf4-50c8049f76e9\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") "
Oct 29 23:26:07.215102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.215024    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-tmp-dir\") pod \"afd19809-a784-4509-bdf4-50c8049f76e9\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") "
Oct 29 23:26:07.215308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.215263    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd19809-a784-4509-bdf4-50c8049f76e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/afd19809-a784-4509-bdf4-50c8049f76e9-tls-certs\") pod \"afd19809-a784-4509-bdf4-50c8049f76e9\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") "
Oct 29 23:26:07.215403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.215384    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd19809-a784-4509-bdf4-50c8049f76e9-kube-api-access-x8wkb\" (UniqueName: \"kubernetes.io/projected/afd19809-a784-4509-bdf4-50c8049f76e9-kube-api-access-x8wkb\") pod \"afd19809-a784-4509-bdf4-50c8049f76e9\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") "
Oct 29 23:26:07.216183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.215452    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-model-cache\") pod \"afd19809-a784-4509-bdf4-50c8049f76e9\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") "
Oct 29 23:26:07.216183 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.215620    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-home\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-home\") pod \"afd19809-a784-4509-bdf4-50c8049f76e9\" (UID: \"afd19809-a784-4509-bdf4-50c8049f76e9\") "
Oct 29 23:26:07.217151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.216895    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-tmp-dir" pod "afd19809-a784-4509-bdf4-50c8049f76e9" (UID: "afd19809-a784-4509-bdf4-50c8049f76e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:07.217265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.217176    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-home" pod "afd19809-a784-4509-bdf4-50c8049f76e9" (UID: "afd19809-a784-4509-bdf4-50c8049f76e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:07.218688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.217400    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-model-cache" pod "afd19809-a784-4509-bdf4-50c8049f76e9" (UID: "afd19809-a784-4509-bdf4-50c8049f76e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:07.221148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.221119    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd19809-a784-4509-bdf4-50c8049f76e9-tls-certs" pod "afd19809-a784-4509-bdf4-50c8049f76e9" (UID: "afd19809-a784-4509-bdf4-50c8049f76e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:26:07.221432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.221401    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd19809-a784-4509-bdf4-50c8049f76e9-kube-api-access-x8wkb" pod "afd19809-a784-4509-bdf4-50c8049f76e9" (UID: "afd19809-a784-4509-bdf4-50c8049f76e9"). InnerVolumeSpecName "kube-api-access-x8wkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:07.234660 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.234012    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-dshm" pod "afd19809-a784-4509-bdf4-50c8049f76e9" (UID: "afd19809-a784-4509-bdf4-50c8049f76e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:07.318066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.318029    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:07.318066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.318061    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:07.318066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.318071    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:07.318322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.318080    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd19809-a784-4509-bdf4-50c8049f76e9-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:07.318322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.318089    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8wkb\" (UniqueName: \"kubernetes.io/projected/afd19809-a784-4509-bdf4-50c8049f76e9-kube-api-access-x8wkb\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:07.318322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.318098    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd19809-a784-4509-bdf4-50c8049f76e9-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:07.596952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.596863    4827 generic.go:356] "Generic (PLEG): container finished" podID="afd19809-a784-4509-bdf4-50c8049f76e9" containerID="97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d" exitCode=0
Oct 29 23:26:07.596952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.596910    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" event={"ID":"afd19809-a784-4509-bdf4-50c8049f76e9","Type":"ContainerDied","Data":"97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d"}
Oct 29 23:26:07.596952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.596934    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p" event={"ID":"afd19809-a784-4509-bdf4-50c8049f76e9","Type":"ContainerDied","Data":"bdecda6e4b8ca95be2f3b6283f806dc4613197ad027014eb4f062d487b53f78d"}
Oct 29 23:26:07.596952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.596930    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"
Oct 29 23:26:07.597553 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.597002    4827 scope.go:122] "RemoveContainer" containerID="97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d"
Oct 29 23:26:07.607874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.607855    4827 scope.go:122] "RemoveContainer" containerID="97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d"
Oct 29 23:26:07.608155 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:07.608135    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d\": container with ID starting with 97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d not found: ID does not exist" containerID="97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d"
Oct 29 23:26:07.608210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.608164    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d"} err="failed to get container status \"97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d\": rpc error: code = NotFound desc = could not find container \"97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d\": container with ID starting with 97a0b97f9a6e630cc587f750a60f0e2d059114e4cac310d182071f23c673167d not found: ID does not exist"
Oct 29 23:26:07.646061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.646035    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"]
Oct 29 23:26:07.656523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:07.656495    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a37090/test-model-a37090-kserve-6cfb57c7dc-lth4p"]
Oct 29 23:26:08.489710 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:08.489675    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" path="/var/lib/kubelet/pods/afd19809-a784-4509-bdf4-50c8049f76e9/volumes"
Oct 29 23:26:13.817819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:13.817328    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:26:15.593427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:15.590854    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"]
Oct 29 23:26:16.643144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.643088    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 29 23:26:16.661789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.656908    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:26:16.712210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.712173    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:26:16.724791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.724732    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:26:16.726115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.726085    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:26:16.727350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.727322    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:26:16.743553 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.743524    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:26:16.744439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.743748    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:26:16.763870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:16.763838    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:26:17.051845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:17.051315    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerName="main" probeResult="failure" output="Get \"https://10.134.1.43:8000/ready\": context deadline exceeded"
Oct 29 23:26:31.459992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.459949    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerName="main" containerID="cri-o://eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675" gracePeriod=45
Oct 29 23:26:31.748911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.748887    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:26:31.821823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.821789    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-dshm\") pod \"df8e24bb-c63d-4658-88fd-14ca4585b952\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") "
Oct 29 23:26:31.822012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.821845    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-tmp-dir\") pod \"df8e24bb-c63d-4658-88fd-14ca4585b952\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") "
Oct 29 23:26:31.822012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.821886    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8e24bb-c63d-4658-88fd-14ca4585b952-tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e24bb-c63d-4658-88fd-14ca4585b952-tls-certs\") pod \"df8e24bb-c63d-4658-88fd-14ca4585b952\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") "
Oct 29 23:26:31.822202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.822174    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8e24bb-c63d-4658-88fd-14ca4585b952-kube-api-access-g756d\" (UniqueName: \"kubernetes.io/projected/df8e24bb-c63d-4658-88fd-14ca4585b952-kube-api-access-g756d\") pod \"df8e24bb-c63d-4658-88fd-14ca4585b952\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") "
Oct 29 23:26:31.822365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.822255    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-home\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-home\") pod \"df8e24bb-c63d-4658-88fd-14ca4585b952\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") "
Oct 29 23:26:31.822365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.822181    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-tmp-dir" pod "df8e24bb-c63d-4658-88fd-14ca4585b952" (UID: "df8e24bb-c63d-4658-88fd-14ca4585b952"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:31.822365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.822312    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-model-cache\") pod \"df8e24bb-c63d-4658-88fd-14ca4585b952\" (UID: \"df8e24bb-c63d-4658-88fd-14ca4585b952\") "
Oct 29 23:26:31.822589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.822567    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-home" pod "df8e24bb-c63d-4658-88fd-14ca4585b952" (UID: "df8e24bb-c63d-4658-88fd-14ca4585b952"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:31.822893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.822870    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-model-cache" pod "df8e24bb-c63d-4658-88fd-14ca4585b952" (UID: "df8e24bb-c63d-4658-88fd-14ca4585b952"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:31.823117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.823101    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:31.823199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.823119    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:31.823199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.823130    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:31.824266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.824239    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-dshm" pod "df8e24bb-c63d-4658-88fd-14ca4585b952" (UID: "df8e24bb-c63d-4658-88fd-14ca4585b952"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:31.824582 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.824558    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8e24bb-c63d-4658-88fd-14ca4585b952-tls-certs" pod "df8e24bb-c63d-4658-88fd-14ca4585b952" (UID: "df8e24bb-c63d-4658-88fd-14ca4585b952"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:26:31.824679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.824613    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8e24bb-c63d-4658-88fd-14ca4585b952-kube-api-access-g756d" pod "df8e24bb-c63d-4658-88fd-14ca4585b952" (UID: "df8e24bb-c63d-4658-88fd-14ca4585b952"). InnerVolumeSpecName "kube-api-access-g756d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:31.924988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.924958    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g756d\" (UniqueName: \"kubernetes.io/projected/df8e24bb-c63d-4658-88fd-14ca4585b952-kube-api-access-g756d\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:31.924988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.924989    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e24bb-c63d-4658-88fd-14ca4585b952-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:31.925184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:31.925004    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e24bb-c63d-4658-88fd-14ca4585b952-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:32.599792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.599282    4827 generic.go:356] "Generic (PLEG): container finished" podID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerID="eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675" exitCode=0
Oct 29 23:26:32.599792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.599607    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" event={"ID":"df8e24bb-c63d-4658-88fd-14ca4585b952","Type":"ContainerDied","Data":"eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675"}
Oct 29 23:26:32.599792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.599674    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq" event={"ID":"df8e24bb-c63d-4658-88fd-14ca4585b952","Type":"ContainerDied","Data":"9811d42c908f5f783654d64166cf3b7a9e0267c02312b0c04cb8c8709e03cee3"}
Oct 29 23:26:32.599792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.599712    4827 scope.go:122] "RemoveContainer" containerID="eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675"
Oct 29 23:26:32.600476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.599940    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"
Oct 29 23:26:32.721203 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.721132    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"]
Oct 29 23:26:32.764590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.763745    4827 scope.go:122] "RemoveContainer" containerID="eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675"
Oct 29 23:26:32.764590 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:32.764249    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675\": container with ID starting with eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675 not found: ID does not exist" containerID="eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675"
Oct 29 23:26:32.764590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.764285    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675"} err="failed to get container status \"eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675\": rpc error: code = NotFound desc = could not find container \"eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675\": container with ID starting with eb47682e2130937518d8a01e2ec4fe3e5f764c829adb66bd18c447b7b6026675 not found: ID does not exist"
Oct 29 23:26:32.784804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:32.775319    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-024aa6/test-model-024aa6-kserve-98c965cd7-9tzrq"]
Oct 29 23:26:33.006849 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:33.006046    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:26:34.490552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:34.490509    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" path="/var/lib/kubelet/pods/df8e24bb-c63d-4658-88fd-14ca4585b952/volumes"
Oct 29 23:26:40.558329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:40.558281    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"]
Oct 29 23:26:40.559231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:40.559176    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy" containerID="cri-o://b1ba41ce76fab7fde919b52d7d56a5b45fda6a25ccc2adc719ba055e4304bc28" gracePeriod=30
Oct 29 23:26:43.281796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.277113    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"]
Oct 29 23:26:43.282312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282203    4827 cpu_manager.go:414] "containerMap: removing container" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main"
Oct 29 23:26:43.282312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282230    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main"
Oct 29 23:26:43.282312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282250    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy"
Oct 29 23:26:43.282312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282257    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy"
Oct 29 23:26:43.282312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282275    4827 cpu_manager.go:414] "containerMap: removing container" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy"
Oct 29 23:26:43.282312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282285    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282333    4827 cpu_manager.go:414] "containerMap: removing container" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerName="main"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282343    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerName="main"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282359    4827 cpu_manager.go:414] "containerMap: removing container" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282367    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282385    4827 cpu_manager.go:414] "containerMap: removing container" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282392    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80096ee0-b2d5-4fac-9a73-6a05813c0ff0" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282562    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd19809-a784-4509-bdf4-50c8049f76e9" containerName="main"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282575    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f572783-4d2e-47bd-a5e2-543d7d1526a5" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282588    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282602    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282614    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8e24bb-c63d-4658-88fd-14ca4585b952" containerName="main"
Oct 29 23:26:43.282653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.282623    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="02be85c9-a940-4c56-a4ce-bcd3fffcc72f" containerName="istio-proxy"
Oct 29 23:26:43.318785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.313970    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"]
Oct 29 23:26:43.318785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.314103    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.352797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.342279    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:26:43.352797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.342398    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:26:43.415061 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.415023    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqqx\" (UniqueName: \"kubernetes.io/projected/4bf0e798-3445-4be0-b791-c0382fee4822-kube-api-access-mqqqx\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.415689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.415641    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.416108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.416085    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.541575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.524460    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.650962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.542579    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.651387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.651359    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqqx\" (UniqueName: \"kubernetes.io/projected/4bf0e798-3445-4be0-b791-c0382fee4822-kube-api-access-mqqqx\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.652149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.652032    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.657792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.657507    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.680785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.678668    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqqx\" (UniqueName: \"kubernetes.io/projected/4bf0e798-3445-4be0-b791-c0382fee4822-kube-api-access-mqqqx\") pod \"maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:43.964624 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:43.964595    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:44.290591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:44.290563    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"]
Oct 29 23:26:44.291452 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:26:44.291426    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf0e798_3445_4be0_b791_c0382fee4822.slice/crio-1eabb6b73d3e41642747f8576271db7711689bf4b8e0c103b20d0a6013c69fdb WatchSource:0}: Error finding container 1eabb6b73d3e41642747f8576271db7711689bf4b8e0c103b20d0a6013c69fdb: Status 404 returned error can't find the container with id 1eabb6b73d3e41642747f8576271db7711689bf4b8e0c103b20d0a6013c69fdb
Oct 29 23:26:44.759826 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:44.759791    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp" event={"ID":"4bf0e798-3445-4be0-b791-c0382fee4822","Type":"ContainerStarted","Data":"1eabb6b73d3e41642747f8576271db7711689bf4b8e0c103b20d0a6013c69fdb"}
Oct 29 23:26:45.783374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:45.783343    4827 generic.go:356] "Generic (PLEG): container finished" podID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerID="b1ba41ce76fab7fde919b52d7d56a5b45fda6a25ccc2adc719ba055e4304bc28" exitCode=0
Oct 29 23:26:45.783831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:45.783409    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" event={"ID":"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e","Type":"ContainerDied","Data":"b1ba41ce76fab7fde919b52d7d56a5b45fda6a25ccc2adc719ba055e4304bc28"}
Oct 29 23:26:45.783831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:45.783472    4827 scope.go:122] "RemoveContainer" containerID="1f409d85d3ffa9b8a286a6ca3c0bfd865fd46f45ae3c91c8f0034a90cd19d2dd"
Oct 29 23:26:45.785526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:45.785493    4827 generic.go:356] "Generic (PLEG): container finished" podID="4bf0e798-3445-4be0-b791-c0382fee4822" containerID="7fd1159b1012cd1340b6b149245bb6c2f25aee0e531c1dac732db2e02e8f7bf0" exitCode=0
Oct 29 23:26:45.785641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:45.785537    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp" event={"ID":"4bf0e798-3445-4be0-b791-c0382fee4822","Type":"ContainerDied","Data":"7fd1159b1012cd1340b6b149245bb6c2f25aee0e531c1dac732db2e02e8f7bf0"}
Oct 29 23:26:45.864933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:45.864908    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:26:46.003430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003390    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-kube-api-access-fvkwm\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-kube-api-access-fvkwm\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.003639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003468    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-token\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-token\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.003639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003509    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-podinfo\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.003853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003644    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-data\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.003853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003680    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istiod-ca-cert\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.003853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003753    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-socket\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.003853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003814    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-certs\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.003853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003855    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-envoy\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.004168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.003891    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-credential-socket\") pod \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\" (UID: \"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e\") "
Oct 29 23:26:46.004392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.004357    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-data" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:46.005529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.004875    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.005529 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.005267    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-certs" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:46.007214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.007176    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istiod-ca-cert" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:26:46.007552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.007526    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-socket" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:46.009425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.009388    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-credential-socket" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:46.011333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.011308    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-token" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:46.043289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.043008    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-kube-api-access-fvkwm" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "kube-api-access-fvkwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:46.069679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.065967    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-podinfo" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:26:46.069679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.066921    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-envoy" pod "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" (UID: "0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:46.105994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.105929    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkwm\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-kube-api-access-fvkwm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.105994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.105952    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.105994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.105962    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.105994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.105970    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.105994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.105979    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.105994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.105987    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.106289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.106001    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.106289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.106012    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:46.794505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.794426    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"
Oct 29 23:26:46.795039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.794427    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq" event={"ID":"0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e","Type":"ContainerDied","Data":"a047dcf4f015d736aebd18f03d856e3be6dd91df102a39775f4bbd30a84bd951"}
Oct 29 23:26:46.795039 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.794567    4827 scope.go:122] "RemoveContainer" containerID="b1ba41ce76fab7fde919b52d7d56a5b45fda6a25ccc2adc719ba055e4304bc28"
Oct 29 23:26:46.860814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.859063    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"]
Oct 29 23:26:46.894270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.879145    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a37090-openshift-default-68fb455bbc-tgtgq"]
Oct 29 23:26:46.985851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:46.985828    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:47.119974 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.119885    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-xks-ca\") pod \"4bf0e798-3445-4be0-b791-c0382fee4822\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") "
Oct 29 23:26:47.120156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.120081    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf0e798-3445-4be0-b791-c0382fee4822-kube-api-access-mqqqx\" (UniqueName: \"kubernetes.io/projected/4bf0e798-3445-4be0-b791-c0382fee4822-kube-api-access-mqqqx\") pod \"4bf0e798-3445-4be0-b791-c0382fee4822\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") "
Oct 29 23:26:47.120156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.120138    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-openshift-ca\") pod \"4bf0e798-3445-4be0-b791-c0382fee4822\" (UID: \"4bf0e798-3445-4be0-b791-c0382fee4822\") "
Oct 29 23:26:47.120486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.120462    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-openshift-ca" pod "4bf0e798-3445-4be0-b791-c0382fee4822" (UID: "4bf0e798-3445-4be0-b791-c0382fee4822"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:26:47.120712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.120691    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:47.122030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.122002    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-xks-ca" pod "4bf0e798-3445-4be0-b791-c0382fee4822" (UID: "4bf0e798-3445-4be0-b791-c0382fee4822"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:26:47.122140 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.122123    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf0e798-3445-4be0-b791-c0382fee4822-kube-api-access-mqqqx" pod "4bf0e798-3445-4be0-b791-c0382fee4822" (UID: "4bf0e798-3445-4be0-b791-c0382fee4822"). InnerVolumeSpecName "kube-api-access-mqqqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:47.221961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.221933    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf0e798-3445-4be0-b791-c0382fee4822-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:47.221961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.221959    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqqqx\" (UniqueName: \"kubernetes.io/projected/4bf0e798-3445-4be0-b791-c0382fee4822-kube-api-access-mqqqx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:47.804414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.804385    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"
Oct 29 23:26:47.804928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.804431    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp" event={"ID":"4bf0e798-3445-4be0-b791-c0382fee4822","Type":"ContainerDied","Data":"1eabb6b73d3e41642747f8576271db7711689bf4b8e0c103b20d0a6013c69fdb"}
Oct 29 23:26:47.804928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:47.804473    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eabb6b73d3e41642747f8576271db7711689bf4b8e0c103b20d0a6013c69fdb"
Oct 29 23:26:48.489478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:48.489444    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" path="/var/lib/kubelet/pods/0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e/volumes"
Oct 29 23:26:51.278612 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:51.278575    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"]
Oct 29 23:26:51.279168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:51.278879    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy" containerID="cri-o://3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552" gracePeriod=30
Oct 29 23:26:56.606802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.606754    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:26:56.683911 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.683877    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-envoy\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684009    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80b345f7-bf4d-4b94-9da2-236481c674db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b345f7-bf4d-4b94-9da2-236481c674db-istio-podinfo\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684042    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-certs\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684085    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b345f7-bf4d-4b94-9da2-236481c674db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b345f7-bf4d-4b94-9da2-236481c674db-istiod-ca-cert\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684282    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-socket\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684322    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-kube-api-access-f8x8r\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-kube-api-access-f8x8r\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684433 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684388    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-istio-token\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-istio-token\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684434    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-credential-socket\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.684488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.684467    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-data\") pod \"80b345f7-bf4d-4b94-9da2-236481c674db\" (UID: \"80b345f7-bf4d-4b94-9da2-236481c674db\") "
Oct 29 23:26:56.685069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.685040    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b345f7-bf4d-4b94-9da2-236481c674db-istiod-ca-cert" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:26:56.685469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.685444    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-data" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:56.685469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.685368    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b345f7-bf4d-4b94-9da2-236481c674db-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.685658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.685639    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-socket" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:56.686373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.686262    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-certs" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:56.686373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.686347    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-credential-socket" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:56.687843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.687807    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-envoy" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:26:56.689895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.689870    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-kube-api-access-f8x8r" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "kube-api-access-f8x8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:56.692022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.691999    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80b345f7-bf4d-4b94-9da2-236481c674db-istio-podinfo" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:26:56.692123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.692003    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-istio-token" pod "80b345f7-bf4d-4b94-9da2-236481c674db" (UID: "80b345f7-bf4d-4b94-9da2-236481c674db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:56.786809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786757    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.786809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786805    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8x8r\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-kube-api-access-f8x8r\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.786809 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786816    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b345f7-bf4d-4b94-9da2-236481c674db-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.787049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786825    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.787049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786835    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.787049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786843    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.787049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786855    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b345f7-bf4d-4b94-9da2-236481c674db-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.787049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.786865    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b345f7-bf4d-4b94-9da2-236481c674db-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:26:56.899244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.899204    4827 generic.go:356] "Generic (PLEG): container finished" podID="80b345f7-bf4d-4b94-9da2-236481c674db" containerID="3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552" exitCode=0
Oct 29 23:26:56.899428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.899268    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"
Oct 29 23:26:56.899428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.899289    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" event={"ID":"80b345f7-bf4d-4b94-9da2-236481c674db","Type":"ContainerDied","Data":"3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552"}
Oct 29 23:26:56.899428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.899341    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj" event={"ID":"80b345f7-bf4d-4b94-9da2-236481c674db","Type":"ContainerDied","Data":"2ec8774388bea7d40a3df1386a572105296c0a65d86abec3aeeb7aaf896002c4"}
Oct 29 23:26:56.899428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.899362    4827 scope.go:122] "RemoveContainer" containerID="3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552"
Oct 29 23:26:56.912365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.912341    4827 scope.go:122] "RemoveContainer" containerID="a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7"
Oct 29 23:26:56.922680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.922660    4827 scope.go:122] "RemoveContainer" containerID="3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552"
Oct 29 23:26:56.923026 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:56.923007    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552\": container with ID starting with 3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552 not found: ID does not exist" containerID="3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552"
Oct 29 23:26:56.923103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.923034    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552"} err="failed to get container status \"3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552\": rpc error: code = NotFound desc = could not find container \"3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552\": container with ID starting with 3831f6bb73d381e2e243450d43f1024144b89771685f5971d2c5f502d0a10552 not found: ID does not exist"
Oct 29 23:26:56.923103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.923054    4827 scope.go:122] "RemoveContainer" containerID="a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7"
Oct 29 23:26:56.923349 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:26:56.923330    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7\": container with ID starting with a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7 not found: ID does not exist" containerID="a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7"
Oct 29 23:26:56.923400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.923355    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7"} err="failed to get container status \"a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7\": rpc error: code = NotFound desc = could not find container \"a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7\": container with ID starting with a126f5576a43b4849ea605fdb0177a952812bf36acc20c0a71280822b1a44cc7 not found: ID does not exist"
Oct 29 23:26:56.949194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.949165    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"]
Oct 29 23:26:56.954208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:56.954182    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-9wpfj"]
Oct 29 23:26:58.492057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:58.492023    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" path="/var/lib/kubelet/pods/80b345f7-bf4d-4b94-9da2-236481c674db/volumes"
Oct 29 23:26:59.596013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.595981    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"]
Oct 29 23:26:59.596637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596623    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4bf0e798-3445-4be0-b791-c0382fee4822" containerName="revoke-keys"
Oct 29 23:26:59.596688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596637    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf0e798-3445-4be0-b791-c0382fee4822" containerName="revoke-keys"
Oct 29 23:26:59.596688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596662    4827 cpu_manager.go:414] "containerMap: removing container" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy"
Oct 29 23:26:59.596688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596667    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547cb68c-1fcb-41c4-944f-246ffafd7f21" containerName="istio-proxy"
Oct 29 23:26:59.596688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596680    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy"
Oct 29 23:26:59.596688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596687    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596696    4827 cpu_manager.go:414] "containerMap: removing container" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596700    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596711    4827 cpu_manager.go:414] "containerMap: removing container" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596716    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596846    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596859    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf0e798-3445-4be0-b791-c0382fee4822" containerName="revoke-keys"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596868    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596875    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy"
Oct 29 23:26:59.596896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.596883    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b345f7-bf4d-4b94-9da2-236481c674db" containerName="istio-proxy"
Oct 29 23:26:59.602115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.602094    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.630985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.630939    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3d24e75-serving-cert\""
Oct 29 23:26:59.633525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.633500    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"]
Oct 29 23:26:59.734418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.734388    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz29w\" (UniqueName: \"kubernetes.io/projected/e87138d4-fac8-4de7-b5d0-021f38532334-kube-api-access-sz29w\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.734653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.734531    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-maas-api-tls\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.734653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.734603    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87138d4-fac8-4de7-b5d0-021f38532334-openshift-service-ca\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.734653 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.734626    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-metrics-certs\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.836015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.835981    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-maas-api-tls\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.836173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.836032    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87138d4-fac8-4de7-b5d0-021f38532334-openshift-service-ca\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.836173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.836053    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-metrics-certs\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.836286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.836253    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz29w\" (UniqueName: \"kubernetes.io/projected/e87138d4-fac8-4de7-b5d0-021f38532334-kube-api-access-sz29w\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.836875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.836850    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87138d4-fac8-4de7-b5d0-021f38532334-openshift-service-ca\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.838573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.838549    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-metrics-certs\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.838666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.838551    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-maas-api-tls\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.904696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.904660    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz29w\" (UniqueName: \"kubernetes.io/projected/e87138d4-fac8-4de7-b5d0-021f38532334-kube-api-access-sz29w\") pod \"maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:26:59.915811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:26:59.915659    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:27:00.275294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:00.275267    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"]
Oct 29 23:27:00.275573 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:27:00.275549    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87138d4_fac8_4de7_b5d0_021f38532334.slice/crio-213e6b8def3261b138c7b4a668dd2062c70353b14c34d70b0e9b7c19f5896227 WatchSource:0}: Error finding container 213e6b8def3261b138c7b4a668dd2062c70353b14c34d70b0e9b7c19f5896227: Status 404 returned error can't find the container with id 213e6b8def3261b138c7b4a668dd2062c70353b14c34d70b0e9b7c19f5896227
Oct 29 23:27:00.276830 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:00.276812    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:27:00.935983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:00.935903    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs" event={"ID":"e87138d4-fac8-4de7-b5d0-021f38532334","Type":"ContainerStarted","Data":"ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123"}
Oct 29 23:27:00.935983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:00.935958    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs" event={"ID":"e87138d4-fac8-4de7-b5d0-021f38532334","Type":"ContainerStarted","Data":"213e6b8def3261b138c7b4a668dd2062c70353b14c34d70b0e9b7c19f5896227"}
Oct 29 23:27:00.936419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:00.936002    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:27:00.978544 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:00.978494    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs" podStartSLOduration=1.5813341840000001 podStartE2EDuration="1.978479348s" totalImagesPullingTime="397.145164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:00.974561558 +0000 UTC m=+2656.597472578" watchObservedRunningTime="2026-10-29 23:27:00.978479348 +0000 UTC m=+2656.601390367"
Oct 29 23:27:02.825043 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:02.825008    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:27:06.987993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:06.987946    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:27:07.486791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.486147    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:27:07.594930 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.594882    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:27:07.626308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.626260    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:27:07.651035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.650991    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:27:07.691821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.685427    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 23:27:07.691821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.687140    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:27:07.742982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.729876    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:27:07.744228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.744192    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 29 23:27:07.809212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:07.808896    4827 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:27:11.151185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:11.151151    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"]
Oct 29 23:27:11.184119 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:11.184087    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"]
Oct 29 23:27:13.573683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:13.573649    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"]
Oct 29 23:27:13.574243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:13.573900    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy" containerID="cri-o://19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308" gracePeriod=30
Oct 29 23:27:16.151893 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:16.151854    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" podUID="40f53fb5-b929-44c4-9e79-9d1258ac3793" containerName="payload-processing" containerID="cri-o://3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc" gracePeriod=25
Oct 29 23:27:16.184664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:16.184624    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" podUID="da18e019-4d40-4d10-923c-db375e21bf0c" containerName="payload-pre-processing" containerID="cri-o://d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067" gracePeriod=25
Oct 29 23:27:16.716074 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:16.716038    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" podUID="40f53fb5-b929-44c4-9e79-9d1258ac3793" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:27:16.723413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:16.723326    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" podUID="da18e019-4d40-4d10-923c-db375e21bf0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:27:18.839643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.839581    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 23:27:18.978741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.978711    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-socket\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.978931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.978895    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-kube-api-access-8blk7\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-kube-api-access-8blk7\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.978945    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-podinfo\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.978976    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-socket" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:18.979092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.978982    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-certs\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979059    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7201b82-d529-4613-bff3-68b8d8fbf266-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7201b82-d529-4613-bff3-68b8d8fbf266-istiod-ca-cert\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979192    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-data\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979218    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-certs" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:18.979278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979226    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-token\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-token\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979286    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-envoy\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979316    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-credential-socket\") pod \"a7201b82-d529-4613-bff3-68b8d8fbf266\" (UID: \"a7201b82-d529-4613-bff3-68b8d8fbf266\") "
Oct 29 23:27:18.979519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979456    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7201b82-d529-4613-bff3-68b8d8fbf266-istiod-ca-cert" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:18.979718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979688    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-data" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:18.979880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.979640    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-credential-socket" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:18.980028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.980007    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:18.980134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.980030    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7201b82-d529-4613-bff3-68b8d8fbf266-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:18.980134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.980043    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:18.980134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.980055    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:18.980134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.980068    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:18.981326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.981303    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-podinfo" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:27:18.981469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.981455    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-envoy" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:18.981627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.981613    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-kube-api-access-8blk7" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "kube-api-access-8blk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:18.981682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:18.981634    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-token" pod "a7201b82-d529-4613-bff3-68b8d8fbf266" (UID: "a7201b82-d529-4613-bff3-68b8d8fbf266"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:19.081376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.081345    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:19.081376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.081374    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:19.081581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.081386    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8blk7\" (UniqueName: \"kubernetes.io/projected/a7201b82-d529-4613-bff3-68b8d8fbf266-kube-api-access-8blk7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:19.081581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.081401    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7201b82-d529-4613-bff3-68b8d8fbf266-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:19.547715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.547680    4827 generic.go:356] "Generic (PLEG): container finished" podID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerID="19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308" exitCode=0
Oct 29 23:27:19.547947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.547744    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"
Oct 29 23:27:19.547947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.547801    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" event={"ID":"a7201b82-d529-4613-bff3-68b8d8fbf266","Type":"ContainerDied","Data":"19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308"}
Oct 29 23:27:19.547947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.547868    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9" event={"ID":"a7201b82-d529-4613-bff3-68b8d8fbf266","Type":"ContainerDied","Data":"39c48962832ba4506f1d59e4af6497c9e12f0828ba747b5e92bda4198ba15f34"}
Oct 29 23:27:19.547947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.547896    4827 scope.go:122] "RemoveContainer" containerID="19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308"
Oct 29 23:27:19.558715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.558696    4827 scope.go:122] "RemoveContainer" containerID="f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b"
Oct 29 23:27:19.567835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.567818    4827 scope.go:122] "RemoveContainer" containerID="19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308"
Oct 29 23:27:19.568125 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:19.568081    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308\": container with ID starting with 19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308 not found: ID does not exist" containerID="19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308"
Oct 29 23:27:19.568125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.568110    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308"} err="failed to get container status \"19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308\": rpc error: code = NotFound desc = could not find container \"19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308\": container with ID starting with 19b3a2a8f8a48478fe527b0cdc024c8529a02899747fb26fb22a663faf950308 not found: ID does not exist"
Oct 29 23:27:19.568234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.568128    4827 scope.go:122] "RemoveContainer" containerID="f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b"
Oct 29 23:27:19.568372 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:19.568356    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b\": container with ID starting with f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b not found: ID does not exist" containerID="f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b"
Oct 29 23:27:19.568410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.568376    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b"} err="failed to get container status \"f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b\": rpc error: code = NotFound desc = could not find container \"f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b\": container with ID starting with f955d70e20b85481e4c186df345b414c003985b15f9e44691a7fb1bf77b95e7b not found: ID does not exist"
Oct 29 23:27:19.607929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.607905    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"]
Oct 29 23:27:19.618108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:19.618085    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-r9vs9"]
Oct 29 23:27:20.489661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:20.489629    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" path="/var/lib/kubelet/pods/a7201b82-d529-4613-bff3-68b8d8fbf266/volumes"
Oct 29 23:27:23.734655 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:23.734618    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"]
Oct 29 23:27:23.744488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:23.744447    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dcacb9-fphlp"]
Oct 29 23:27:24.491128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:24.491095    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf0e798-3445-4be0-b791-c0382fee4822" path="/var/lib/kubelet/pods/4bf0e798-3445-4be0-b791-c0382fee4822/volumes"
Oct 29 23:27:25.181558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.181525    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"]
Oct 29 23:27:25.182112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.181810    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp" podUID="b9bc9ee5-382e-4008-815d-e697f95f8699" containerName="maas-api" containerID="cri-o://f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e" gracePeriod=30
Oct 29 23:27:25.191934 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:25.191897    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bc9ee5_382e_4008_815d_e697f95f8699.slice/crio-f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:27:25.571822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.566021    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:27:25.709591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.709484    4827 generic.go:356] "Generic (PLEG): container finished" podID="b9bc9ee5-382e-4008-815d-e697f95f8699" containerID="f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e" exitCode=0
Oct 29 23:27:25.710200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.710167    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"
Oct 29 23:27:25.710920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.710891    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp" event={"ID":"b9bc9ee5-382e-4008-815d-e697f95f8699","Type":"ContainerDied","Data":"f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e"}
Oct 29 23:27:25.711153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.711126    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp" event={"ID":"b9bc9ee5-382e-4008-815d-e697f95f8699","Type":"ContainerDied","Data":"b11f5f7d332d79186fa0a144627236039cd6dd0233373d18d63169dc73769ca3"}
Oct 29 23:27:25.711303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.711290    4827 scope.go:122] "RemoveContainer" containerID="f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e"
Oct 29 23:27:25.735414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.735382    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bc9ee5-382e-4008-815d-e697f95f8699-kube-api-access-5qhnq\" (UniqueName: \"kubernetes.io/projected/b9bc9ee5-382e-4008-815d-e697f95f8699-kube-api-access-5qhnq\") pod \"b9bc9ee5-382e-4008-815d-e697f95f8699\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") "
Oct 29 23:27:25.764873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.742295    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-metrics-certs\") pod \"b9bc9ee5-382e-4008-815d-e697f95f8699\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") "
Oct 29 23:27:25.766423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.766398    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-maas-api-tls\") pod \"b9bc9ee5-382e-4008-815d-e697f95f8699\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") "
Oct 29 23:27:25.767525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.767504    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bc9ee5-382e-4008-815d-e697f95f8699-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9ee5-382e-4008-815d-e697f95f8699-openshift-service-ca\") pod \"b9bc9ee5-382e-4008-815d-e697f95f8699\" (UID: \"b9bc9ee5-382e-4008-815d-e697f95f8699\") "
Oct 29 23:27:25.791788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.774050    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bc9ee5-382e-4008-815d-e697f95f8699-openshift-service-ca" pod "b9bc9ee5-382e-4008-815d-e697f95f8699" (UID: "b9bc9ee5-382e-4008-815d-e697f95f8699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:25.877807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.835992    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bc9ee5-382e-4008-815d-e697f95f8699-kube-api-access-5qhnq" pod "b9bc9ee5-382e-4008-815d-e697f95f8699" (UID: "b9bc9ee5-382e-4008-815d-e697f95f8699"). InnerVolumeSpecName "kube-api-access-5qhnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:25.877807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.876553    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9ee5-382e-4008-815d-e697f95f8699-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:25.877807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.876584    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qhnq\" (UniqueName: \"kubernetes.io/projected/b9bc9ee5-382e-4008-815d-e697f95f8699-kube-api-access-5qhnq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:25.892002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.891739    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-metrics-certs" pod "b9bc9ee5-382e-4008-815d-e697f95f8699" (UID: "b9bc9ee5-382e-4008-815d-e697f95f8699"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:27:25.892002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.891896    4827 scope.go:122] "RemoveContainer" containerID="f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e"
Oct 29 23:27:25.894099 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:25.893989    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e\": container with ID starting with f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e not found: ID does not exist" containerID="f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e"
Oct 29 23:27:25.894099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.894034    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e"} err="failed to get container status \"f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e\": rpc error: code = NotFound desc = could not find container \"f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e\": container with ID starting with f7ccb9bd3c7d5b56919ef14d9e22c28788ddbd35a66456a55df9d3333ebb5c0e not found: ID does not exist"
Oct 29 23:27:25.894452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.894414    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-maas-api-tls" pod "b9bc9ee5-382e-4008-815d-e697f95f8699" (UID: "b9bc9ee5-382e-4008-815d-e697f95f8699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:27:25.978403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.978115    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:25.978403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:25.978147    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9ee5-382e-4008-815d-e697f95f8699-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:26.055384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:26.055269    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"]
Oct 29 23:27:26.067680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:26.067651    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-024aa6-77858c5469-rh2gp"]
Oct 29 23:27:26.491137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:26.491106    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bc9ee5-382e-4008-815d-e697f95f8699" path="/var/lib/kubelet/pods/b9bc9ee5-382e-4008-815d-e697f95f8699/volumes"
Oct 29 23:27:26.699985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:26.699954    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" podUID="40f53fb5-b929-44c4-9e79-9d1258ac3793" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:27:26.699985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:26.699964    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" podUID="da18e019-4d40-4d10-923c-db375e21bf0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:27:31.956137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956102    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"]
Oct 29 23:27:31.956866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956845    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy"
Oct 29 23:27:31.956958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956868    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy"
Oct 29 23:27:31.956958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956898    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy"
Oct 29 23:27:31.956958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956903    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8d31ff-dfdf-4393-bfa3-4d8ba63abd0e" containerName="istio-proxy"
Oct 29 23:27:31.956958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956919    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b9bc9ee5-382e-4008-815d-e697f95f8699" containerName="maas-api"
Oct 29 23:27:31.956958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956924    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bc9ee5-382e-4008-815d-e697f95f8699" containerName="maas-api"
Oct 29 23:27:31.956958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956933    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy"
Oct 29 23:27:31.956958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.956939    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy"
Oct 29 23:27:31.957332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.957068    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bc9ee5-382e-4008-815d-e697f95f8699" containerName="maas-api"
Oct 29 23:27:31.957332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.957076    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy"
Oct 29 23:27:31.962076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.962052    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:31.980372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:31.980347    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"]
Oct 29 23:27:32.020962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.020932    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:27:32.020962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.020932    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:27:32.088621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.088586    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.088808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.088626    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.088808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.088732    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597cx\" (UniqueName: \"kubernetes.io/projected/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-kube-api-access-597cx\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.190651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.190618    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.190842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.190661    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.190842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.190716    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597cx\" (UniqueName: \"kubernetes.io/projected/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-kube-api-access-597cx\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.191707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.191676    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.193190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.193172    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.212438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.212379    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-597cx\" (UniqueName: \"kubernetes.io/projected/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-kube-api-access-597cx\") pod \"maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.278407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.278375    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:32.660133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.660108    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"]
Oct 29 23:27:32.663165 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:27:32.663135    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd228ff83_5e37_4f2e_ac1c_30d5790a2d30.slice/crio-8fd93405a8eaea6d48ae074f439e1de9f2ede98429deb88e6eed0f8e19a0adcc WatchSource:0}: Error finding container 8fd93405a8eaea6d48ae074f439e1de9f2ede98429deb88e6eed0f8e19a0adcc: Status 404 returned error can't find the container with id 8fd93405a8eaea6d48ae074f439e1de9f2ede98429deb88e6eed0f8e19a0adcc
Oct 29 23:27:32.769753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:32.769717    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg" event={"ID":"d228ff83-5e37-4f2e-ac1c-30d5790a2d30","Type":"ContainerStarted","Data":"8fd93405a8eaea6d48ae074f439e1de9f2ede98429deb88e6eed0f8e19a0adcc"}
Oct 29 23:27:32.825673 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:32.825638    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:27:33.777942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:33.777910    4827 generic.go:356] "Generic (PLEG): container finished" podID="d228ff83-5e37-4f2e-ac1c-30d5790a2d30" containerID="e06fb27ac17a5e4813ba83a65b6a606664bd0df9eb04a8cf035189349aa0db63" exitCode=0
Oct 29 23:27:33.778321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:33.777992    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg" event={"ID":"d228ff83-5e37-4f2e-ac1c-30d5790a2d30","Type":"ContainerDied","Data":"e06fb27ac17a5e4813ba83a65b6a606664bd0df9eb04a8cf035189349aa0db63"}
Oct 29 23:27:34.580816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:34.579389    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"]
Oct 29 23:27:34.580816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:34.579712    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" containerID="cri-o://b88e3ad478fe8a50b38795e5973c45130d901abfbf1f392e1aabdfc6ae751180" gracePeriod=30
Oct 29 23:27:35.108773 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.108737    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:35.184578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.184544    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-kube-api-access-597cx\" (UniqueName: \"kubernetes.io/projected/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-kube-api-access-597cx\") pod \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") "
Oct 29 23:27:35.184807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.184705    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-openshift-ca\") pod \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") "
Oct 29 23:27:35.184807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.184760    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-xks-ca\") pod \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\" (UID: \"d228ff83-5e37-4f2e-ac1c-30d5790a2d30\") "
Oct 29 23:27:35.185137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.185112    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-openshift-ca" pod "d228ff83-5e37-4f2e-ac1c-30d5790a2d30" (UID: "d228ff83-5e37-4f2e-ac1c-30d5790a2d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:35.185563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.185529    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:35.188054 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.188015    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-xks-ca" pod "d228ff83-5e37-4f2e-ac1c-30d5790a2d30" (UID: "d228ff83-5e37-4f2e-ac1c-30d5790a2d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:27:35.188195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.188158    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-kube-api-access-597cx" pod "d228ff83-5e37-4f2e-ac1c-30d5790a2d30" (UID: "d228ff83-5e37-4f2e-ac1c-30d5790a2d30"). InnerVolumeSpecName "kube-api-access-597cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:35.286697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.286620    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-597cx\" (UniqueName: \"kubernetes.io/projected/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-kube-api-access-597cx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:35.286697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.286649    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d228ff83-5e37-4f2e-ac1c-30d5790a2d30-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:35.813341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.813313    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"
Oct 29 23:27:35.813341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.813336    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg" event={"ID":"d228ff83-5e37-4f2e-ac1c-30d5790a2d30","Type":"ContainerDied","Data":"8fd93405a8eaea6d48ae074f439e1de9f2ede98429deb88e6eed0f8e19a0adcc"}
Oct 29 23:27:35.813574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:35.813385    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd93405a8eaea6d48ae074f439e1de9f2ede98429deb88e6eed0f8e19a0adcc"
Oct 29 23:27:36.478048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.478022    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:27:36.511606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.511583    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:27:36.604693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.604661    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f53fb5-b929-44c4-9e79-9d1258ac3793-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f53fb5-b929-44c4-9e79-9d1258ac3793-plugins-config-volume\") pod \"40f53fb5-b929-44c4-9e79-9d1258ac3793\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") "
Oct 29 23:27:36.604907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.604803    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da18e019-4d40-4d10-923c-db375e21bf0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da18e019-4d40-4d10-923c-db375e21bf0c-plugins-config-volume\") pod \"da18e019-4d40-4d10-923c-db375e21bf0c\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") "
Oct 29 23:27:36.604992 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.604914    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da18e019-4d40-4d10-923c-db375e21bf0c-kube-api-access-2s5ld\" (UniqueName: \"kubernetes.io/projected/da18e019-4d40-4d10-923c-db375e21bf0c-kube-api-access-2s5ld\") pod \"da18e019-4d40-4d10-923c-db375e21bf0c\" (UID: \"da18e019-4d40-4d10-923c-db375e21bf0c\") "
Oct 29 23:27:36.605111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.605092    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f53fb5-b929-44c4-9e79-9d1258ac3793-kube-api-access-bjfb9\" (UniqueName: \"kubernetes.io/projected/40f53fb5-b929-44c4-9e79-9d1258ac3793-kube-api-access-bjfb9\") pod \"40f53fb5-b929-44c4-9e79-9d1258ac3793\" (UID: \"40f53fb5-b929-44c4-9e79-9d1258ac3793\") "
Oct 29 23:27:36.605190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.605172    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f53fb5-b929-44c4-9e79-9d1258ac3793-plugins-config-volume" pod "40f53fb5-b929-44c4-9e79-9d1258ac3793" (UID: "40f53fb5-b929-44c4-9e79-9d1258ac3793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:36.605249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.605199    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da18e019-4d40-4d10-923c-db375e21bf0c-plugins-config-volume" pod "da18e019-4d40-4d10-923c-db375e21bf0c" (UID: "da18e019-4d40-4d10-923c-db375e21bf0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:36.605637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.605609    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f53fb5-b929-44c4-9e79-9d1258ac3793-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:36.605637 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.605630    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da18e019-4d40-4d10-923c-db375e21bf0c-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:36.607159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.607140    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f53fb5-b929-44c4-9e79-9d1258ac3793-kube-api-access-bjfb9" pod "40f53fb5-b929-44c4-9e79-9d1258ac3793" (UID: "40f53fb5-b929-44c4-9e79-9d1258ac3793"). InnerVolumeSpecName "kube-api-access-bjfb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:36.607255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.607198    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da18e019-4d40-4d10-923c-db375e21bf0c-kube-api-access-2s5ld" pod "da18e019-4d40-4d10-923c-db375e21bf0c" (UID: "da18e019-4d40-4d10-923c-db375e21bf0c"). InnerVolumeSpecName "kube-api-access-2s5ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:36.707450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.707372    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s5ld\" (UniqueName: \"kubernetes.io/projected/da18e019-4d40-4d10-923c-db375e21bf0c-kube-api-access-2s5ld\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:36.707450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.707403    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfb9\" (UniqueName: \"kubernetes.io/projected/40f53fb5-b929-44c4-9e79-9d1258ac3793-kube-api-access-bjfb9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:36.822377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.822344    4827 generic.go:356] "Generic (PLEG): container finished" podID="da18e019-4d40-4d10-923c-db375e21bf0c" containerID="d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067" exitCode=0
Oct 29 23:27:36.822595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.822395    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"
Oct 29 23:27:36.822595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.822424    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" event={"ID":"da18e019-4d40-4d10-923c-db375e21bf0c","Type":"ContainerDied","Data":"d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067"}
Oct 29 23:27:36.822595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.822480    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6" event={"ID":"da18e019-4d40-4d10-923c-db375e21bf0c","Type":"ContainerDied","Data":"15cf2c5e6cf8b0e8da5bbb39581d00f7de3bc63c0ef86893a94e96bf76bee6dd"}
Oct 29 23:27:36.822595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.822507    4827 scope.go:122] "RemoveContainer" containerID="d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067"
Oct 29 23:27:36.824702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.824677    4827 generic.go:356] "Generic (PLEG): container finished" podID="40f53fb5-b929-44c4-9e79-9d1258ac3793" containerID="3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc" exitCode=0
Oct 29 23:27:36.824836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.824729    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"
Oct 29 23:27:36.824836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.824793    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" event={"ID":"40f53fb5-b929-44c4-9e79-9d1258ac3793","Type":"ContainerDied","Data":"3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc"}
Oct 29 23:27:36.824836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.824831    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk" event={"ID":"40f53fb5-b929-44c4-9e79-9d1258ac3793","Type":"ContainerDied","Data":"0eb1fb94d9afa25912cb073ed52d0eb848089852364b84b644063e59f8aeb086"}
Oct 29 23:27:36.840677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.840654    4827 scope.go:122] "RemoveContainer" containerID="d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067"
Oct 29 23:27:36.840990 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:36.840970    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067\": container with ID starting with d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067 not found: ID does not exist" containerID="d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067"
Oct 29 23:27:36.841089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.840998    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067"} err="failed to get container status \"d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067\": rpc error: code = NotFound desc = could not find container \"d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067\": container with ID starting with d475cffee60e4dc9bb9b41b167f5e26b28c3ae37fdac1f5efe2fc64ea5e75067 not found: ID does not exist"
Oct 29 23:27:36.841089 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.841015    4827 scope.go:122] "RemoveContainer" containerID="3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc"
Oct 29 23:27:36.850749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.850725    4827 scope.go:122] "RemoveContainer" containerID="3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc"
Oct 29 23:27:36.851218 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:27:36.851139    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc\": container with ID starting with 3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc not found: ID does not exist" containerID="3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc"
Oct 29 23:27:36.851313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.851227    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc"} err="failed to get container status \"3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc\": rpc error: code = NotFound desc = could not find container \"3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc\": container with ID starting with 3f87137ff075f2ecf931659301078a41084266a8835e6e0edd115bd0ea3862fc not found: ID does not exist"
Oct 29 23:27:36.886314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.886288    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"]
Oct 29 23:27:36.893777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.893744    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-32717a-5df98bb7-vgcf6"]
Oct 29 23:27:36.972850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.971911    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"]
Oct 29 23:27:36.987923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:36.977292    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-32717a-797965448d-p8zhk"]
Oct 29 23:27:37.625346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:37.625313    4827 patch_prober.go:29] interesting pod/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:27:37.625743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:37.625371    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:27:38.488374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:38.488337    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f53fb5-b929-44c4-9e79-9d1258ac3793" path="/var/lib/kubelet/pods/40f53fb5-b929-44c4-9e79-9d1258ac3793/volumes"
Oct 29 23:27:38.488898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:38.488876    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da18e019-4d40-4d10-923c-db375e21bf0c" path="/var/lib/kubelet/pods/da18e019-4d40-4d10-923c-db375e21bf0c/volumes"
Oct 29 23:27:39.850888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.850815    4827 generic.go:356] "Generic (PLEG): container finished" podID="6c041e2d-bd10-460e-a154-06946afd4220" containerID="b88e3ad478fe8a50b38795e5973c45130d901abfbf1f392e1aabdfc6ae751180" exitCode=0
Oct 29 23:27:39.851271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.850895    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" event={"ID":"6c041e2d-bd10-460e-a154-06946afd4220","Type":"ContainerDied","Data":"b88e3ad478fe8a50b38795e5973c45130d901abfbf1f392e1aabdfc6ae751180"}
Oct 29 23:27:39.851271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.850981    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd" event={"ID":"6c041e2d-bd10-460e-a154-06946afd4220","Type":"ContainerDied","Data":"544e70eb1fc6d35007d8f4cc70b6046593b91124ed911e6e6fea97e7a3d82218"}
Oct 29 23:27:39.851271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.851009    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="544e70eb1fc6d35007d8f4cc70b6046593b91124ed911e6e6fea97e7a3d82218"
Oct 29 23:27:39.851271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.851035    4827 scope.go:122] "RemoveContainer" containerID="b04db3035bfab780619b1bbacdd4b8fe1ff9b78252b07a8160be3947af93996a"
Oct 29 23:27:39.865090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.865069    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:27:39.994426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.993668    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-istio-token\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-istio-token\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.994649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.994519    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c041e2d-bd10-460e-a154-06946afd4220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c041e2d-bd10-460e-a154-06946afd4220-istio-podinfo\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.995185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.995160    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-socket\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.995311 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.995213    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-certs\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.995391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.995325    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-data\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.995785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.995680    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-envoy\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.995890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.995855    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-kube-api-access-jmvql\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-kube-api-access-jmvql\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.996049 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.996024    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c041e2d-bd10-460e-a154-06946afd4220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c041e2d-bd10-460e-a154-06946afd4220-istiod-ca-cert\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:39.996138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.996101    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-credential-socket\") pod \"6c041e2d-bd10-460e-a154-06946afd4220\" (UID: \"6c041e2d-bd10-460e-a154-06946afd4220\") "
Oct 29 23:27:40.000465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.996701    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-data" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:40.000465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:39.999255    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-certs" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:40.004441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.002276    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c041e2d-bd10-460e-a154-06946afd4220-istiod-ca-cert" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:40.004441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.002348    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c041e2d-bd10-460e-a154-06946afd4220-istio-podinfo" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:27:40.004441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.002690    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-credential-socket" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:40.004441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.002835    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-istio-token" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:40.004441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.003974    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-envoy" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:40.004940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.004903    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-socket" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:27:40.007857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.007835    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c041e2d-bd10-460e-a154-06946afd4220-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.007997    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008119 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.008106    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.008246    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c041e2d-bd10-460e-a154-06946afd4220-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.008423    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.008539    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.008640    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.008734    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c041e2d-bd10-460e-a154-06946afd4220-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.008957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.007917    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-kube-api-access-jmvql" pod "6c041e2d-bd10-460e-a154-06946afd4220" (UID: "6c041e2d-bd10-460e-a154-06946afd4220"). InnerVolumeSpecName "kube-api-access-jmvql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:40.109829 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.109741    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvql\" (UniqueName: \"kubernetes.io/projected/6c041e2d-bd10-460e-a154-06946afd4220-kube-api-access-jmvql\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:40.859385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.859311    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"
Oct 29 23:27:40.950080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.950044    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"]
Oct 29 23:27:40.960938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:40.960900    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-024aa6-openshift-default-7b647c645c-gfzxd"]
Oct 29 23:27:42.490404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:42.490371    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c041e2d-bd10-460e-a154-06946afd4220" path="/var/lib/kubelet/pods/6c041e2d-bd10-460e-a154-06946afd4220/volumes"
Oct 29 23:27:44.539743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:44.539711    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"]
Oct 29 23:27:44.540278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:44.540048    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl" podUID="ae64c2b6-7120-4267-86bd-b3d8e6140252" containerName="maas-api" containerID="cri-o://82833d08df761b27bd7d9bc62fce776c8d6d453e2a279bf52ef4f751bc095cb1" gracePeriod=30
Oct 29 23:27:44.958580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:44.958162    4827 generic.go:356] "Generic (PLEG): container finished" podID="ae64c2b6-7120-4267-86bd-b3d8e6140252" containerID="82833d08df761b27bd7d9bc62fce776c8d6d453e2a279bf52ef4f751bc095cb1" exitCode=0
Oct 29 23:27:44.958580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:44.958457    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl" event={"ID":"ae64c2b6-7120-4267-86bd-b3d8e6140252","Type":"ContainerDied","Data":"82833d08df761b27bd7d9bc62fce776c8d6d453e2a279bf52ef4f751bc095cb1"}
Oct 29 23:27:44.958580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:44.958516    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl" event={"ID":"ae64c2b6-7120-4267-86bd-b3d8e6140252","Type":"ContainerDied","Data":"0a88acab14165bdf9968a30a2f65eb89cae4fbb8a6db4a529559c4ef9b6f85d2"}
Oct 29 23:27:44.958580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:44.958540    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a88acab14165bdf9968a30a2f65eb89cae4fbb8a6db4a529559c4ef9b6f85d2"
Oct 29 23:27:44.966247 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:44.965906    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:27:45.023793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.021302    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-maas-api-tls\") pod \"ae64c2b6-7120-4267-86bd-b3d8e6140252\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") "
Oct 29 23:27:45.023793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.021664    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-metrics-certs\") pod \"ae64c2b6-7120-4267-86bd-b3d8e6140252\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") "
Oct 29 23:27:45.023793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.021988    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae64c2b6-7120-4267-86bd-b3d8e6140252-kube-api-access-nhfrt\" (UniqueName: \"kubernetes.io/projected/ae64c2b6-7120-4267-86bd-b3d8e6140252-kube-api-access-nhfrt\") pod \"ae64c2b6-7120-4267-86bd-b3d8e6140252\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") "
Oct 29 23:27:45.023793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.022998    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae64c2b6-7120-4267-86bd-b3d8e6140252-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae64c2b6-7120-4267-86bd-b3d8e6140252-openshift-service-ca\") pod \"ae64c2b6-7120-4267-86bd-b3d8e6140252\" (UID: \"ae64c2b6-7120-4267-86bd-b3d8e6140252\") "
Oct 29 23:27:45.025115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.025075    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae64c2b6-7120-4267-86bd-b3d8e6140252-openshift-service-ca" pod "ae64c2b6-7120-4267-86bd-b3d8e6140252" (UID: "ae64c2b6-7120-4267-86bd-b3d8e6140252"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:45.061011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.044170    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae64c2b6-7120-4267-86bd-b3d8e6140252-kube-api-access-nhfrt" pod "ae64c2b6-7120-4267-86bd-b3d8e6140252" (UID: "ae64c2b6-7120-4267-86bd-b3d8e6140252"). InnerVolumeSpecName "kube-api-access-nhfrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:45.061011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.059510    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-maas-api-tls" pod "ae64c2b6-7120-4267-86bd-b3d8e6140252" (UID: "ae64c2b6-7120-4267-86bd-b3d8e6140252"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:27:45.066305 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.066247    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-metrics-certs" pod "ae64c2b6-7120-4267-86bd-b3d8e6140252" (UID: "ae64c2b6-7120-4267-86bd-b3d8e6140252"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:27:45.176633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.176237    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfrt\" (UniqueName: \"kubernetes.io/projected/ae64c2b6-7120-4267-86bd-b3d8e6140252-kube-api-access-nhfrt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:45.176633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.176278    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae64c2b6-7120-4267-86bd-b3d8e6140252-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:45.176633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.176294    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:45.176633 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.176307    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae64c2b6-7120-4267-86bd-b3d8e6140252-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:45.965628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:45.965596    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"
Oct 29 23:27:46.017002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:46.016972    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"]
Oct 29 23:27:46.028216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:46.028187    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-32717a-55f5567694-l7ktl"]
Oct 29 23:27:46.488812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:46.488783    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae64c2b6-7120-4267-86bd-b3d8e6140252" path="/var/lib/kubelet/pods/ae64c2b6-7120-4267-86bd-b3d8e6140252/volumes"
Oct 29 23:27:47.323423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:47.323400    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:27:47.444251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:47.444225    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:27:51.179788 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:51.171975    4827 scope.go:122] "RemoveContainer" containerID="82833d08df761b27bd7d9bc62fce776c8d6d453e2a279bf52ef4f751bc095cb1"
Oct 29 23:27:55.174092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174057    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"]
Oct 29 23:27:55.174691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174677    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ae64c2b6-7120-4267-86bd-b3d8e6140252" containerName="maas-api"
Oct 29 23:27:55.174737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174692    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae64c2b6-7120-4267-86bd-b3d8e6140252" containerName="maas-api"
Oct 29 23:27:55.174737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174713    4827 cpu_manager.go:414] "containerMap: removing container" podUID="da18e019-4d40-4d10-923c-db375e21bf0c" containerName="payload-pre-processing"
Oct 29 23:27:55.174737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174718    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18e019-4d40-4d10-923c-db375e21bf0c" containerName="payload-pre-processing"
Oct 29 23:27:55.174737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174731    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy"
Oct 29 23:27:55.174737 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174736    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174742    4827 cpu_manager.go:414] "containerMap: removing container" podUID="40f53fb5-b929-44c4-9e79-9d1258ac3793" containerName="payload-processing"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174747    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f53fb5-b929-44c4-9e79-9d1258ac3793" containerName="payload-processing"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174760    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d228ff83-5e37-4f2e-ac1c-30d5790a2d30" containerName="revoke-keys"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174780    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d228ff83-5e37-4f2e-ac1c-30d5790a2d30" containerName="revoke-keys"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174906    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174919    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d228ff83-5e37-4f2e-ac1c-30d5790a2d30" containerName="revoke-keys"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174931    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae64c2b6-7120-4267-86bd-b3d8e6140252" containerName="maas-api"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174940    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="da18e019-4d40-4d10-923c-db375e21bf0c" containerName="payload-pre-processing"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174949    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f53fb5-b929-44c4-9e79-9d1258ac3793" containerName="payload-processing"
Oct 29 23:27:55.174984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.174957    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7201b82-d529-4613-bff3-68b8d8fbf266" containerName="istio-proxy"
Oct 29 23:27:55.180284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.180263    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.207889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.207856    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"]
Oct 29 23:27:55.214589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.214566    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:27:55.214735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.214617    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:27:55.250837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.250760    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.251029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.251005    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64js\" (UniqueName: \"kubernetes.io/projected/4a00b195-91a7-4a01-b505-550dc6692924-kube-api-access-p64js\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.251232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.251213    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.352551 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.352510    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.352757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.352622    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64js\" (UniqueName: \"kubernetes.io/projected/4a00b195-91a7-4a01-b505-550dc6692924-kube-api-access-p64js\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.352870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.352775    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.353275 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.353231    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.355139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.355117    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.374807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.374755    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64js\" (UniqueName: \"kubernetes.io/projected/4a00b195-91a7-4a01-b505-550dc6692924-kube-api-access-p64js\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.495013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.494937    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:55.798704 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:27:55.798664    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a00b195_91a7_4a01_b505_550dc6692924.slice/crio-c9dc51259e1a69d1f607530a144bea753a4ad10021292655c0bee6185f22f6c9 WatchSource:0}: Error finding container c9dc51259e1a69d1f607530a144bea753a4ad10021292655c0bee6185f22f6c9: Status 404 returned error can't find the container with id c9dc51259e1a69d1f607530a144bea753a4ad10021292655c0bee6185f22f6c9
Oct 29 23:27:55.798858 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:55.798836    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"]
Oct 29 23:27:56.064816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:56.064710    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p" event={"ID":"4a00b195-91a7-4a01-b505-550dc6692924","Type":"ContainerStarted","Data":"c9dc51259e1a69d1f607530a144bea753a4ad10021292655c0bee6185f22f6c9"}
Oct 29 23:27:57.073483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:57.073406    4827 generic.go:356] "Generic (PLEG): container finished" podID="4a00b195-91a7-4a01-b505-550dc6692924" containerID="f38bef2f930b9e0701e06ba845fdd34f4b89988108625001a8d7a002c6f45a3b" exitCode=0
Oct 29 23:27:57.073876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:57.073488    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p" event={"ID":"4a00b195-91a7-4a01-b505-550dc6692924","Type":"ContainerDied","Data":"f38bef2f930b9e0701e06ba845fdd34f4b89988108625001a8d7a002c6f45a3b"}
Oct 29 23:27:58.231877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.231854    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:58.276500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.276475    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-xks-ca\") pod \"4a00b195-91a7-4a01-b505-550dc6692924\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") "
Oct 29 23:27:58.276659 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.276627    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-openshift-ca\") pod \"4a00b195-91a7-4a01-b505-550dc6692924\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") "
Oct 29 23:27:58.276738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.276710    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a00b195-91a7-4a01-b505-550dc6692924-kube-api-access-p64js\" (UniqueName: \"kubernetes.io/projected/4a00b195-91a7-4a01-b505-550dc6692924-kube-api-access-p64js\") pod \"4a00b195-91a7-4a01-b505-550dc6692924\" (UID: \"4a00b195-91a7-4a01-b505-550dc6692924\") "
Oct 29 23:27:58.277028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.277002    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-openshift-ca" pod "4a00b195-91a7-4a01-b505-550dc6692924" (UID: "4a00b195-91a7-4a01-b505-550dc6692924"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:58.277290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.277240    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:58.278836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.278805    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a00b195-91a7-4a01-b505-550dc6692924-kube-api-access-p64js" pod "4a00b195-91a7-4a01-b505-550dc6692924" (UID: "4a00b195-91a7-4a01-b505-550dc6692924"). InnerVolumeSpecName "kube-api-access-p64js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:58.278955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.278910    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-xks-ca" pod "4a00b195-91a7-4a01-b505-550dc6692924" (UID: "4a00b195-91a7-4a01-b505-550dc6692924"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:27:58.378517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.378481    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64js\" (UniqueName: \"kubernetes.io/projected/4a00b195-91a7-4a01-b505-550dc6692924-kube-api-access-p64js\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:58.378517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:58.378512    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a00b195-91a7-4a01-b505-550dc6692924-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:27:59.089120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:59.089088    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"
Oct 29 23:27:59.089313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:59.089092    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p" event={"ID":"4a00b195-91a7-4a01-b505-550dc6692924","Type":"ContainerDied","Data":"c9dc51259e1a69d1f607530a144bea753a4ad10021292655c0bee6185f22f6c9"}
Oct 29 23:27:59.089313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:27:59.089239    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9dc51259e1a69d1f607530a144bea753a4ad10021292655c0bee6185f22f6c9"
Oct 29 23:28:01.776385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:01.776348    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"]
Oct 29 23:28:01.776871 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:01.776619    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f" podUID="b969954a-8f9f-4a42-9684-2853d025a1fc" containerName="maas-api" containerID="cri-o://0a886643e704824e3957693d140ea1c9f45904eba53a96f264aabd117a3beb81" gracePeriod=30
Oct 29 23:28:02.373262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.373222    4827 generic.go:356] "Generic (PLEG): container finished" podID="b969954a-8f9f-4a42-9684-2853d025a1fc" containerID="0a886643e704824e3957693d140ea1c9f45904eba53a96f264aabd117a3beb81" exitCode=0
Oct 29 23:28:02.373797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.373753    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f" event={"ID":"b969954a-8f9f-4a42-9684-2853d025a1fc","Type":"ContainerDied","Data":"0a886643e704824e3957693d140ea1c9f45904eba53a96f264aabd117a3beb81"}
Oct 29 23:28:02.399457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.399088    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:28:02.438898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.438865    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-metrics-certs\") pod \"b969954a-8f9f-4a42-9684-2853d025a1fc\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") "
Oct 29 23:28:02.439040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.438937    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b969954a-8f9f-4a42-9684-2853d025a1fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b969954a-8f9f-4a42-9684-2853d025a1fc-openshift-service-ca\") pod \"b969954a-8f9f-4a42-9684-2853d025a1fc\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") "
Oct 29 23:28:02.439040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.438980    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b969954a-8f9f-4a42-9684-2853d025a1fc-kube-api-access-85qm6\" (UniqueName: \"kubernetes.io/projected/b969954a-8f9f-4a42-9684-2853d025a1fc-kube-api-access-85qm6\") pod \"b969954a-8f9f-4a42-9684-2853d025a1fc\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") "
Oct 29 23:28:02.439179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.439163    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-maas-api-tls\") pod \"b969954a-8f9f-4a42-9684-2853d025a1fc\" (UID: \"b969954a-8f9f-4a42-9684-2853d025a1fc\") "
Oct 29 23:28:02.439383 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.439350    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b969954a-8f9f-4a42-9684-2853d025a1fc-openshift-service-ca" pod "b969954a-8f9f-4a42-9684-2853d025a1fc" (UID: "b969954a-8f9f-4a42-9684-2853d025a1fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:02.439600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.439574    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b969954a-8f9f-4a42-9684-2853d025a1fc-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:02.441749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.441390    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-metrics-certs" pod "b969954a-8f9f-4a42-9684-2853d025a1fc" (UID: "b969954a-8f9f-4a42-9684-2853d025a1fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:02.441892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.441782    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-maas-api-tls" pod "b969954a-8f9f-4a42-9684-2853d025a1fc" (UID: "b969954a-8f9f-4a42-9684-2853d025a1fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:02.442209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.442187    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b969954a-8f9f-4a42-9684-2853d025a1fc-kube-api-access-85qm6" pod "b969954a-8f9f-4a42-9684-2853d025a1fc" (UID: "b969954a-8f9f-4a42-9684-2853d025a1fc"). InnerVolumeSpecName "kube-api-access-85qm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:02.540492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.540457    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:02.540492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.540485    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b969954a-8f9f-4a42-9684-2853d025a1fc-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:02.540677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:02.540498    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85qm6\" (UniqueName: \"kubernetes.io/projected/b969954a-8f9f-4a42-9684-2853d025a1fc-kube-api-access-85qm6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:02.826573 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:02.826539    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:28:03.382015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:03.381985    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"
Oct 29 23:28:03.382328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:03.382020    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f" event={"ID":"b969954a-8f9f-4a42-9684-2853d025a1fc","Type":"ContainerDied","Data":"7e9cbec4a16a3a7ff957c7d164621d6a84c8c57d46881836433252a835861a23"}
Oct 29 23:28:03.382328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:03.382086    4827 scope.go:122] "RemoveContainer" containerID="0a886643e704824e3957693d140ea1c9f45904eba53a96f264aabd117a3beb81"
Oct 29 23:28:03.434701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:03.434671    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"]
Oct 29 23:28:03.445047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:03.445020    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f"]
Oct 29 23:28:04.524787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:04.523464    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b969954a-8f9f-4a42-9684-2853d025a1fc" path="/var/lib/kubelet/pods/b969954a-8f9f-4a42-9684-2853d025a1fc/volumes"
Oct 29 23:28:04.771060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:04.771021    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"]
Oct 29 23:28:04.781278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:04.781205    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc6328-ftzgg"]
Oct 29 23:28:05.114813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:05.114720    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c880c7-866fc4bc84-qg74f" podUID="b969954a-8f9f-4a42-9684-2853d025a1fc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.254:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:06.573791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:06.557852    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d228ff83-5e37-4f2e-ac1c-30d5790a2d30" path="/var/lib/kubelet/pods/d228ff83-5e37-4f2e-ac1c-30d5790a2d30/volumes"
Oct 29 23:28:10.743991 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.743956    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"]
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744581    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b969954a-8f9f-4a42-9684-2853d025a1fc" containerName="maas-api"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744594    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b969954a-8f9f-4a42-9684-2853d025a1fc" containerName="maas-api"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744610    4827 cpu_manager.go:414] "containerMap: removing container" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744615    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744624    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4a00b195-91a7-4a01-b505-550dc6692924" containerName="revoke-keys"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744629    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a00b195-91a7-4a01-b505-550dc6692924" containerName="revoke-keys"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744731    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a00b195-91a7-4a01-b505-550dc6692924" containerName="revoke-keys"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744738    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c041e2d-bd10-460e-a154-06946afd4220" containerName="istio-proxy"
Oct 29 23:28:10.746352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.744752    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b969954a-8f9f-4a42-9684-2853d025a1fc" containerName="maas-api"
Oct 29 23:28:10.747884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.747868    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.766754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.766728    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"]
Oct 29 23:28:10.799065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.799036    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:28:10.799235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.799116    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:28:10.876162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.876132    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.876343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.876253    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.876411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.876389    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclkm\" (UniqueName: \"kubernetes.io/projected/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-kube-api-access-gclkm\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.977709 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.977683    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.977923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.977823    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.977972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.977942    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gclkm\" (UniqueName: \"kubernetes.io/projected/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-kube-api-access-gclkm\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.978335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.978311    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.980350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.980322    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:10.999659 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:10.999608    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclkm\" (UniqueName: \"kubernetes.io/projected/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-kube-api-access-gclkm\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:11.062961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:11.062935    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:11.366835 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:28:11.366802    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e29cd45_78a1_4a57_845f_e576ed6fdfcf.slice/crio-f5d95756cbb73c2048cfd136e311e2e8f2ab7fd312417ee0d9650030306aad6b WatchSource:0}: Error finding container f5d95756cbb73c2048cfd136e311e2e8f2ab7fd312417ee0d9650030306aad6b: Status 404 returned error can't find the container with id f5d95756cbb73c2048cfd136e311e2e8f2ab7fd312417ee0d9650030306aad6b
Oct 29 23:28:11.367068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:11.367045    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"]
Oct 29 23:28:11.641697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:11.641655    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg" event={"ID":"5e29cd45-78a1-4a57-845f-e576ed6fdfcf","Type":"ContainerStarted","Data":"f5d95756cbb73c2048cfd136e311e2e8f2ab7fd312417ee0d9650030306aad6b"}
Oct 29 23:28:12.389535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:12.389503    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"]
Oct 29 23:28:12.390186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:12.390154    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy" containerID="cri-o://31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3" gracePeriod=30
Oct 29 23:28:12.693720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:12.693678    4827 generic.go:356] "Generic (PLEG): container finished" podID="5e29cd45-78a1-4a57-845f-e576ed6fdfcf" containerID="05250d6e9f5b065c89289ebcb55cfa1de92c3ae543d3d56574940a9eefa7dfc7" exitCode=0
Oct 29 23:28:12.693923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:12.693732    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg" event={"ID":"5e29cd45-78a1-4a57-845f-e576ed6fdfcf","Type":"ContainerDied","Data":"05250d6e9f5b065c89289ebcb55cfa1de92c3ae543d3d56574940a9eefa7dfc7"}
Oct 29 23:28:13.871084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:13.871062    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:14.037619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.037534    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-xks-ca\") pod \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") "
Oct 29 23:28:14.037805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.037621    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-kube-api-access-gclkm\" (UniqueName: \"kubernetes.io/projected/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-kube-api-access-gclkm\") pod \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") "
Oct 29 23:28:14.037805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.037721    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-openshift-ca\") pod \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\" (UID: \"5e29cd45-78a1-4a57-845f-e576ed6fdfcf\") "
Oct 29 23:28:14.038111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.038087    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-openshift-ca" pod "5e29cd45-78a1-4a57-845f-e576ed6fdfcf" (UID: "5e29cd45-78a1-4a57-845f-e576ed6fdfcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:14.038401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.038376    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.039616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.039590    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-xks-ca" pod "5e29cd45-78a1-4a57-845f-e576ed6fdfcf" (UID: "5e29cd45-78a1-4a57-845f-e576ed6fdfcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:14.039793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.039752    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-kube-api-access-gclkm" pod "5e29cd45-78a1-4a57-845f-e576ed6fdfcf" (UID: "5e29cd45-78a1-4a57-845f-e576ed6fdfcf"). InnerVolumeSpecName "kube-api-access-gclkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:14.139615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.139581    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gclkm\" (UniqueName: \"kubernetes.io/projected/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-kube-api-access-gclkm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.139615 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.139612    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e29cd45-78a1-4a57-845f-e576ed6fdfcf-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.708248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.708221    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"
Oct 29 23:28:14.708412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.708269    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg" event={"ID":"5e29cd45-78a1-4a57-845f-e576ed6fdfcf","Type":"ContainerDied","Data":"f5d95756cbb73c2048cfd136e311e2e8f2ab7fd312417ee0d9650030306aad6b"}
Oct 29 23:28:14.708412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:14.708314    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d95756cbb73c2048cfd136e311e2e8f2ab7fd312417ee0d9650030306aad6b"
Oct 29 23:28:17.673806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.673703    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:28:17.732286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.732251    4827 generic.go:356] "Generic (PLEG): container finished" podID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerID="31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3" exitCode=0
Oct 29 23:28:17.732903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.732313    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"
Oct 29 23:28:17.732903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.732341    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" event={"ID":"fef9cfdd-9924-4e7e-8de1-37510651061a","Type":"ContainerDied","Data":"31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3"}
Oct 29 23:28:17.732903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.732395    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7" event={"ID":"fef9cfdd-9924-4e7e-8de1-37510651061a","Type":"ContainerDied","Data":"aa1f27e93b0ca3e1edaa560198a92722937e36d61f5e331a6471b1c859ad06f6"}
Oct 29 23:28:17.732903 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.732426    4827 scope.go:122] "RemoveContainer" containerID="31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3"
Oct 29 23:28:17.753909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.752263    4827 scope.go:122] "RemoveContainer" containerID="31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3"
Oct 29 23:28:17.753909 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:17.752599    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3\": container with ID starting with 31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3 not found: ID does not exist" containerID="31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3"
Oct 29 23:28:17.753909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.752628    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3"} err="failed to get container status \"31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3\": rpc error: code = NotFound desc = could not find container \"31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3\": container with ID starting with 31144c0f34395c475bb7971b4bce663892f5abaafd4b05125b968c332040f6f3 not found: ID does not exist"
Oct 29 23:28:17.799029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.798996    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-socket\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799045    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-podinfo\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799071    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-certs\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799131    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-token\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-token\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799199 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799157    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-envoy\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799248    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-kube-api-access-58ljt\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-kube-api-access-58ljt\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799267    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-socket" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:17.799427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799349    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-data\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799391    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef9cfdd-9924-4e7e-8de1-37510651061a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef9cfdd-9924-4e7e-8de1-37510651061a-istiod-ca-cert\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799427 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799406    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-certs" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:17.799698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799458    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-credential-socket\") pod \"fef9cfdd-9924-4e7e-8de1-37510651061a\" (UID: \"fef9cfdd-9924-4e7e-8de1-37510651061a\") "
Oct 29 23:28:17.799698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799554    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-data" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:17.799942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799900    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-credential-socket" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:17.799942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.799827    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef9cfdd-9924-4e7e-8de1-37510651061a-istiod-ca-cert" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:17.800240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.800219    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.800284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.800247    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.800284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.800264    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.800284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.800277    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef9cfdd-9924-4e7e-8de1-37510651061a-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.800382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.800292    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.801401 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.801377    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-token" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:17.801549 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.801533    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-kube-api-access-58ljt" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "kube-api-access-58ljt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:17.801781 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.801744    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-podinfo" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:28:17.801863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.801741    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-envoy" pod "fef9cfdd-9924-4e7e-8de1-37510651061a" (UID: "fef9cfdd-9924-4e7e-8de1-37510651061a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:17.901181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.901152    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.901181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.901177    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.901181 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.901186    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef9cfdd-9924-4e7e-8de1-37510651061a-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:17.901405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:17.901195    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ljt\" (UniqueName: \"kubernetes.io/projected/fef9cfdd-9924-4e7e-8de1-37510651061a-kube-api-access-58ljt\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:18.080382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:18.080345    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"]
Oct 29 23:28:18.083865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:18.083841    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-pzvf7"]
Oct 29 23:28:18.488861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:18.488828    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" path="/var/lib/kubelet/pods/fef9cfdd-9924-4e7e-8de1-37510651061a/volumes"
Oct 29 23:28:28.137913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.134552    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"]
Oct 29 23:28:28.137913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.134816    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9" podUID="9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" containerName="maas-api" containerID="cri-o://61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0" gracePeriod=30
Oct 29 23:28:28.535622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.535594    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:28:28.604388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.604345    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-metrics-certs\") pod \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") "
Oct 29 23:28:28.604600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.604539    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-kube-api-access-bsm7d\" (UniqueName: \"kubernetes.io/projected/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-kube-api-access-bsm7d\") pod \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") "
Oct 29 23:28:28.604600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.604580    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls\") pod \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") "
Oct 29 23:28:28.604750 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.604654    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-openshift-service-ca\") pod \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\" (UID: \"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315\") "
Oct 29 23:28:28.619359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.618580    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-openshift-service-ca" pod "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" (UID: "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:28.705295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.676953    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-metrics-certs" pod "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" (UID: "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:28.718792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.706314    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls" pod "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" (UID: "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:28.721177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.720367    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:28.721177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.720404    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:28.721177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.720419    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:28.729878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.729823    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-kube-api-access-bsm7d" pod "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" (UID: "9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315"). InnerVolumeSpecName "kube-api-access-bsm7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:28.812424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.812329    4827 generic.go:356] "Generic (PLEG): container finished" podID="9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" containerID="61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0" exitCode=0
Oct 29 23:28:28.812424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.812375    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9" event={"ID":"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315","Type":"ContainerDied","Data":"61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0"}
Oct 29 23:28:28.812424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.812394    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"
Oct 29 23:28:28.812716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.812431    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9" event={"ID":"9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315","Type":"ContainerDied","Data":"a480534bbc3bdfb26fef7f31d40b3a969f0c5b5383c2144325b46d0af692d696"}
Oct 29 23:28:28.812716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.812460    4827 scope.go:122] "RemoveContainer" containerID="61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0"
Oct 29 23:28:28.821898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.821875    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsm7d\" (UniqueName: \"kubernetes.io/projected/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315-kube-api-access-bsm7d\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:28.823677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.823656    4827 scope.go:122] "RemoveContainer" containerID="61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0"
Oct 29 23:28:28.823987 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:28.823965    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0\": container with ID starting with 61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0 not found: ID does not exist" containerID="61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0"
Oct 29 23:28:28.824051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.823993    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0"} err="failed to get container status \"61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0\": rpc error: code = NotFound desc = could not find container \"61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0\": container with ID starting with 61ed5a7f8df8c6360cb9fdf540eda44fcdd271d403654df0f54ecb29a3f092b0 not found: ID does not exist"
Oct 29 23:28:28.860618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.860581    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"]
Oct 29 23:28:28.864532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:28.864505    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b8f5c4-86857cd767-cgsm9"]
Oct 29 23:28:29.180120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:29.180087    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"]
Oct 29 23:28:29.183296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:29.183268    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b8f5c4-lxw9p"]
Oct 29 23:28:30.493979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:30.493949    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a00b195-91a7-4a01-b505-550dc6692924" path="/var/lib/kubelet/pods/4a00b195-91a7-4a01-b505-550dc6692924/volumes"
Oct 29 23:28:30.494754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:30.494732    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" path="/var/lib/kubelet/pods/9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315/volumes"
Oct 29 23:28:32.823864 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:32.823827    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:28:36.296790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.296335    4827 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x7gtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:28:36.296790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.296439    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x7gtt" podUID="d56dd7a0-03e2-4206-a490-555d93824666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:36.708024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.707959    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:36.731891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.731847    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:36.745285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.745251    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:36.765318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.765283    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:36.766352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.766325    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:36.767261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.767238    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 29 23:28:36.863801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.860976    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 29 23:28:36.880056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:36.878902    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb" podUID="fabbc053-4c55-47e4-8345-b2e049215d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 23:28:37.162118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.162059    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"]
Oct 29 23:28:37.163228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.163195    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy" containerID="cri-o://b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac" gracePeriod=30
Oct 29 23:28:37.611849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.611752    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"]
Oct 29 23:28:37.612378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612363    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5e29cd45-78a1-4a57-845f-e576ed6fdfcf" containerName="revoke-keys"
Oct 29 23:28:37.612424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612380    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e29cd45-78a1-4a57-845f-e576ed6fdfcf" containerName="revoke-keys"
Oct 29 23:28:37.612424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612404    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy"
Oct 29 23:28:37.612424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612409    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy"
Oct 29 23:28:37.612523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612428    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" containerName="maas-api"
Oct 29 23:28:37.612523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612433    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" containerName="maas-api"
Oct 29 23:28:37.612592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612536    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e29cd45-78a1-4a57-845f-e576ed6fdfcf" containerName="revoke-keys"
Oct 29 23:28:37.612592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612548    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef9cfdd-9924-4e7e-8de1-37510651061a" containerName="istio-proxy"
Oct 29 23:28:37.612592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.612557    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e10d1eb-2a88-4cbe-a085-9dbf9ef7c315" containerName="maas-api"
Oct 29 23:28:37.617425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.617407    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.634862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.634839    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:28:37.634862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.634844    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:28:37.638613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.638591    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"]
Oct 29 23:28:37.708647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.708610    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxplm\" (UniqueName: \"kubernetes.io/projected/90006c25-90cb-49b2-9570-740ac953fa05-kube-api-access-lxplm\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.708839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.708748    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.708946 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.708928    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.810182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.810149    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.810395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.810250    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxplm\" (UniqueName: \"kubernetes.io/projected/90006c25-90cb-49b2-9570-740ac953fa05-kube-api-access-lxplm\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.810395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.810325    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.810955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.810923    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.812578 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.812562    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.831798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.831745    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxplm\" (UniqueName: \"kubernetes.io/projected/90006c25-90cb-49b2-9570-740ac953fa05-kube-api-access-lxplm\") pod \"maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:37.933673 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:37.933641    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:38.258432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:38.258403    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"]
Oct 29 23:28:38.260023 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:28:38.260000    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90006c25_90cb_49b2_9570_740ac953fa05.slice/crio-99249be60722f6e06c90017bfe53ed25748116d47add2e1ade8f47a6cbe78b73 WatchSource:0}: Error finding container 99249be60722f6e06c90017bfe53ed25748116d47add2e1ade8f47a6cbe78b73: Status 404 returned error can't find the container with id 99249be60722f6e06c90017bfe53ed25748116d47add2e1ade8f47a6cbe78b73
Oct 29 23:28:38.474272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:38.474241    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"]
Oct 29 23:28:38.474495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:38.474471    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy" containerID="cri-o://8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6" gracePeriod=30
Oct 29 23:28:39.167717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.167677    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn" event={"ID":"90006c25-90cb-49b2-9570-740ac953fa05","Type":"ContainerStarted","Data":"addaa7b867d8075bffdff0b0a14bb8c1217a34c6b2298becfc4fa7db191b6d69"}
Oct 29 23:28:39.168215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.167734    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn" event={"ID":"90006c25-90cb-49b2-9570-740ac953fa05","Type":"ContainerStarted","Data":"99249be60722f6e06c90017bfe53ed25748116d47add2e1ade8f47a6cbe78b73"}
Oct 29 23:28:39.208509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.208456    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn" podStartSLOduration=2.208438104 podStartE2EDuration="2.208438104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:39.204730357 +0000 UTC m=+2754.827641376" watchObservedRunningTime="2026-10-29 23:28:39.208438104 +0000 UTC m=+2754.831349123"
Oct 29 23:28:39.235315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.235280    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"]
Oct 29 23:28:39.239377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.239357    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.259336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.259307    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"]
Oct 29 23:28:39.451564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.451477    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.451738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.451716    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.451814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.451781    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9kr\" (UniqueName: \"kubernetes.io/projected/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-kube-api-access-xb9kr\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.553400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.553369    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.553601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.553421    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9kr\" (UniqueName: \"kubernetes.io/projected/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-kube-api-access-xb9kr\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.553601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.553495    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.554193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.554168    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.555808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.555786    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.575165 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.575138    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9kr\" (UniqueName: \"kubernetes.io/projected/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-kube-api-access-xb9kr\") pod \"maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:39.855350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:39.855268    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:40.159439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:40.159412    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"]
Oct 29 23:28:40.160190 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:28:40.160157    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5c492a_c8aa_444f_af7e_e906c4f7b4cc.slice/crio-80d8bdef16dac492358c830113dfdea839a33b1392e5fcd81aa5d0fd30e2c2cb WatchSource:0}: Error finding container 80d8bdef16dac492358c830113dfdea839a33b1392e5fcd81aa5d0fd30e2c2cb: Status 404 returned error can't find the container with id 80d8bdef16dac492358c830113dfdea839a33b1392e5fcd81aa5d0fd30e2c2cb
Oct 29 23:28:40.175895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:40.175863    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv" event={"ID":"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc","Type":"ContainerStarted","Data":"80d8bdef16dac492358c830113dfdea839a33b1392e5fcd81aa5d0fd30e2c2cb"}
Oct 29 23:28:40.178365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:40.178229    4827 generic.go:356] "Generic (PLEG): container finished" podID="90006c25-90cb-49b2-9570-740ac953fa05" containerID="addaa7b867d8075bffdff0b0a14bb8c1217a34c6b2298becfc4fa7db191b6d69" exitCode=0
Oct 29 23:28:40.178365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:40.178293    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn" event={"ID":"90006c25-90cb-49b2-9570-740ac953fa05","Type":"ContainerDied","Data":"addaa7b867d8075bffdff0b0a14bb8c1217a34c6b2298becfc4fa7db191b6d69"}
Oct 29 23:28:41.211968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.211931    4827 generic.go:356] "Generic (PLEG): container finished" podID="fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" containerID="9883ff9726cdfff36592e34f094135f7a0c849a4f82e21dac2797881940babd2" exitCode=0
Oct 29 23:28:41.213220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.213190    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv" event={"ID":"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc","Type":"ContainerDied","Data":"9883ff9726cdfff36592e34f094135f7a0c849a4f82e21dac2797881940babd2"}
Oct 29 23:28:41.475515 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.475488    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:41.626832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.626735    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-xks-ca\") pod \"90006c25-90cb-49b2-9570-740ac953fa05\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") "
Oct 29 23:28:41.627000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.626846    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90006c25-90cb-49b2-9570-740ac953fa05-kube-api-access-lxplm\" (UniqueName: \"kubernetes.io/projected/90006c25-90cb-49b2-9570-740ac953fa05-kube-api-access-lxplm\") pod \"90006c25-90cb-49b2-9570-740ac953fa05\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") "
Oct 29 23:28:41.627000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.626879    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-openshift-ca\") pod \"90006c25-90cb-49b2-9570-740ac953fa05\" (UID: \"90006c25-90cb-49b2-9570-740ac953fa05\") "
Oct 29 23:28:41.627310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.627284    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-openshift-ca" pod "90006c25-90cb-49b2-9570-740ac953fa05" (UID: "90006c25-90cb-49b2-9570-740ac953fa05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:41.627685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.627664    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:41.629500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.629478    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-xks-ca" pod "90006c25-90cb-49b2-9570-740ac953fa05" (UID: "90006c25-90cb-49b2-9570-740ac953fa05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:41.629566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.629507    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90006c25-90cb-49b2-9570-740ac953fa05-kube-api-access-lxplm" pod "90006c25-90cb-49b2-9570-740ac953fa05" (UID: "90006c25-90cb-49b2-9570-740ac953fa05"). InnerVolumeSpecName "kube-api-access-lxplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:41.652577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.652548    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"]
Oct 29 23:28:41.655848 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.655824    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5e52e0-mwkfg"]
Oct 29 23:28:41.728591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.728560    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90006c25-90cb-49b2-9570-740ac953fa05-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:41.728591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:41.728587    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxplm\" (UniqueName: \"kubernetes.io/projected/90006c25-90cb-49b2-9570-740ac953fa05-kube-api-access-lxplm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.221097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.221057    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn" event={"ID":"90006c25-90cb-49b2-9570-740ac953fa05","Type":"ContainerDied","Data":"99249be60722f6e06c90017bfe53ed25748116d47add2e1ade8f47a6cbe78b73"}
Oct 29 23:28:42.221526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.221121    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99249be60722f6e06c90017bfe53ed25748116d47add2e1ade8f47a6cbe78b73"
Oct 29 23:28:42.221526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.221072    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"
Oct 29 23:28:42.404606 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.404571    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"]
Oct 29 23:28:42.408975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.407908    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:42.480692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.480661    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:28:42.491087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.491057    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e29cd45-78a1-4a57-845f-e576ed6fdfcf" path="/var/lib/kubelet/pods/5e29cd45-78a1-4a57-845f-e576ed6fdfcf/volumes"
Oct 29 23:28:42.538533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.537817    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-openshift-ca\") pod \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") "
Oct 29 23:28:42.538533 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.537980    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-kube-api-access-xb9kr\" (UniqueName: \"kubernetes.io/projected/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-kube-api-access-xb9kr\") pod \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") "
Oct 29 23:28:42.539197 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.538575    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-xks-ca\") pod \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\" (UID: \"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc\") "
Oct 29 23:28:42.540834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.540809    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-openshift-ca" pod "fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" (UID: "fb5c492a-c8aa-444f-af7e-e906c4f7b4cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:42.540993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.540872    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-kube-api-access-xb9kr" pod "fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" (UID: "fb5c492a-c8aa-444f-af7e-e906c4f7b4cc"). InnerVolumeSpecName "kube-api-access-xb9kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:42.545011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.544988    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-xks-ca" pod "fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" (UID: "fb5c492a-c8aa-444f-af7e-e906c4f7b4cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:42.640927 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.640896    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2d52c78-f846-4674-9556-28940d3e6c13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d52c78-f846-4674-9556-28940d3e6c13-istio-podinfo\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.640944    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-envoy\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641120    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-istio-token\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-istio-token\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641196    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-socket\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641258    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-credential-socket\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641309    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d52c78-f846-4674-9556-28940d3e6c13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d52c78-f846-4674-9556-28940d3e6c13-istiod-ca-cert\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641342    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-certs\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641375    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-kube-api-access-kkd2j\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-kube-api-access-kkd2j\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641429 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641412    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-data\") pod \"c2d52c78-f846-4674-9556-28940d3e6c13\" (UID: \"c2d52c78-f846-4674-9556-28940d3e6c13\") "
Oct 29 23:28:42.641590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641505    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-socket" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:42.641730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641713    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-certs" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:42.641812 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641735    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d52c78-f846-4674-9556-28940d3e6c13-istiod-ca-cert" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:42.641970 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.641948    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-data" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:42.642134 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642117    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-credential-socket" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:42.642225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642191    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.642225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642215    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb9kr\" (UniqueName: \"kubernetes.io/projected/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-kube-api-access-xb9kr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.642358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642233    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.642358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642249    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d52c78-f846-4674-9556-28940d3e6c13-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.642358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642263    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.642358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642275    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.642358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.642288    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.643534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.643513    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-envoy" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:42.644268 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.644246    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2d52c78-f846-4674-9556-28940d3e6c13-istio-podinfo" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:28:42.645042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.645017    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-istio-token" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:42.645610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.645587    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-kube-api-access-kkd2j" pod "c2d52c78-f846-4674-9556-28940d3e6c13" (UID: "c2d52c78-f846-4674-9556-28940d3e6c13"). InnerVolumeSpecName "kube-api-access-kkd2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:42.744263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.744169    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d52c78-f846-4674-9556-28940d3e6c13-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.744263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.744196    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.744263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.744207    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.744263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.744220    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d52c78-f846-4674-9556-28940d3e6c13-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:42.744263 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:42.744230    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkd2j\" (UniqueName: \"kubernetes.io/projected/c2d52c78-f846-4674-9556-28940d3e6c13-kube-api-access-kkd2j\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.237029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.236998    4827 generic.go:356] "Generic (PLEG): container finished" podID="c2d52c78-f846-4674-9556-28940d3e6c13" containerID="b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac" exitCode=0
Oct 29 23:28:43.237472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.237054    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"
Oct 29 23:28:43.237472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.237082    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" event={"ID":"c2d52c78-f846-4674-9556-28940d3e6c13","Type":"ContainerDied","Data":"b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac"}
Oct 29 23:28:43.237472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.237137    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9" event={"ID":"c2d52c78-f846-4674-9556-28940d3e6c13","Type":"ContainerDied","Data":"3f5ca50683e3c417c8ee6789b883dddfc9a8b50870873f994769c1a570d4930d"}
Oct 29 23:28:43.237472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.237167    4827 scope.go:122] "RemoveContainer" containerID="b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac"
Oct 29 23:28:43.238647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.238625    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv" event={"ID":"fb5c492a-c8aa-444f-af7e-e906c4f7b4cc","Type":"ContainerDied","Data":"80d8bdef16dac492358c830113dfdea839a33b1392e5fcd81aa5d0fd30e2c2cb"}
Oct 29 23:28:43.238749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.238655    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"
Oct 29 23:28:43.238749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.238665    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d8bdef16dac492358c830113dfdea839a33b1392e5fcd81aa5d0fd30e2c2cb"
Oct 29 23:28:43.248670 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.248651    4827 scope.go:122] "RemoveContainer" containerID="cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069"
Oct 29 23:28:43.258446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.258429    4827 scope.go:122] "RemoveContainer" containerID="b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac"
Oct 29 23:28:43.258712 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:43.258681    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac\": container with ID starting with b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac not found: ID does not exist" containerID="b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac"
Oct 29 23:28:43.258785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.258720    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac"} err="failed to get container status \"b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac\": rpc error: code = NotFound desc = could not find container \"b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac\": container with ID starting with b13ef43cd3068f5158f485c2f12501a5c0cb6b2bc2dcea940808b9548f201eac not found: ID does not exist"
Oct 29 23:28:43.258785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.258737    4827 scope.go:122] "RemoveContainer" containerID="cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069"
Oct 29 23:28:43.258990 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:43.258971    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069\": container with ID starting with cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069 not found: ID does not exist" containerID="cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069"
Oct 29 23:28:43.259031 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.258996    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069"} err="failed to get container status \"cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069\": rpc error: code = NotFound desc = could not find container \"cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069\": container with ID starting with cdbc004bbbcf90ce0772604dc1342d9334d615394e8b9bdc11ceb61d5f35c069 not found: ID does not exist"
Oct 29 23:28:43.298236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.298198    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"]
Oct 29 23:28:43.308861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.308833    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755453-openshift-default-7d9b8f4b45-hwzg9"]
Oct 29 23:28:43.816741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.816717    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:28:43.873825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.873790    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-kube-api-access-2fjbx\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-kube-api-access-2fjbx\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.873848    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-socket\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.873910    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-credential-socket\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.873945    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79663068-a8f3-4945-bcc6-fba2936cd3c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79663068-a8f3-4945-bcc6-fba2936cd3c5-istiod-ca-cert\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.874036    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-certs\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.874145    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-data\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874225 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.874201    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-podinfo\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.874279    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-token\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-token\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.874396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.874315    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-envoy\") pod \"79663068-a8f3-4945-bcc6-fba2936cd3c5\" (UID: \"79663068-a8f3-4945-bcc6-fba2936cd3c5\") "
Oct 29 23:28:43.875094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.875049    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-socket" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:43.875360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.875303    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-certs" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:43.875472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.875373    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-data" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:43.875724 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.875700    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79663068-a8f3-4945-bcc6-fba2936cd3c5-istiod-ca-cert" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:43.875896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.875756    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-credential-socket" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:43.877569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.877542    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-token" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:43.877688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.877640    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-podinfo" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:28:43.877954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.877936    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-kube-api-access-2fjbx" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "kube-api-access-2fjbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:43.879162 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.879141    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-envoy" pod "79663068-a8f3-4945-bcc6-fba2936cd3c5" (UID: "79663068-a8f3-4945-bcc6-fba2936cd3c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:43.976352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976321    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976349    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976359    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjbx\" (UniqueName: \"kubernetes.io/projected/79663068-a8f3-4945-bcc6-fba2936cd3c5-kube-api-access-2fjbx\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976369    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976379    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976388    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79663068-a8f3-4945-bcc6-fba2936cd3c5-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976396    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976404    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:43.976619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:43.976413    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79663068-a8f3-4945-bcc6-fba2936cd3c5-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:44.248394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.248357    4827 generic.go:356] "Generic (PLEG): container finished" podID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerID="8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6" exitCode=0
Oct 29 23:28:44.248850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.248408    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"
Oct 29 23:28:44.248850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.248408    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" event={"ID":"79663068-a8f3-4945-bcc6-fba2936cd3c5","Type":"ContainerDied","Data":"8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6"}
Oct 29 23:28:44.248850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.248530    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg" event={"ID":"79663068-a8f3-4945-bcc6-fba2936cd3c5","Type":"ContainerDied","Data":"c889f9e8cd28ff03cd0b055c28a110b890afe079a09f958f45b9b767327a3314"}
Oct 29 23:28:44.248850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.248555    4827 scope.go:122] "RemoveContainer" containerID="8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6"
Oct 29 23:28:44.259954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.259916    4827 scope.go:122] "RemoveContainer" containerID="06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099"
Oct 29 23:28:44.269081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.269063    4827 scope.go:122] "RemoveContainer" containerID="8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6"
Oct 29 23:28:44.269316 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:44.269296    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6\": container with ID starting with 8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6 not found: ID does not exist" containerID="8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6"
Oct 29 23:28:44.269384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.269325    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6"} err="failed to get container status \"8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6\": rpc error: code = NotFound desc = could not find container \"8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6\": container with ID starting with 8ed17ee2e74b9f74bf1e3f7e5230fed52e8ccd7308bdb9e7961949a1593342a6 not found: ID does not exist"
Oct 29 23:28:44.269384 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.269346    4827 scope.go:122] "RemoveContainer" containerID="06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099"
Oct 29 23:28:44.269569 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:44.269550    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099\": container with ID starting with 06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099 not found: ID does not exist" containerID="06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099"
Oct 29 23:28:44.269625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.269578    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099"} err="failed to get container status \"06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099\": rpc error: code = NotFound desc = could not find container \"06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099\": container with ID starting with 06fad7072e6928c1d2193559c93162f6e8d64be41bd524792260d3a0f6dad099 not found: ID does not exist"
Oct 29 23:28:44.297410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.297386    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"]
Oct 29 23:28:44.308281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.308258    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b8f5c4-openshift-default-5ccb974469-fmrlg"]
Oct 29 23:28:44.490308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.490279    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" path="/var/lib/kubelet/pods/79663068-a8f3-4945-bcc6-fba2936cd3c5/volumes"
Oct 29 23:28:44.490808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:44.490791    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" path="/var/lib/kubelet/pods/c2d52c78-f846-4674-9556-28940d3e6c13/volumes"
Oct 29 23:28:48.093066 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:48.093035    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"]
Oct 29 23:28:49.401808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:49.401700    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"]
Oct 29 23:28:49.402376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:49.402090    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy" containerID="cri-o://ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4" gracePeriod=30
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.246245    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"]
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247205    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247229    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247248    4827 cpu_manager.go:414] "containerMap: removing container" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247256    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247302    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" containerName="revoke-keys"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247311    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" containerName="revoke-keys"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247329    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247337    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247380    4827 cpu_manager.go:414] "containerMap: removing container" podUID="90006c25-90cb-49b2-9570-740ac953fa05" containerName="revoke-keys"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247387    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90006c25-90cb-49b2-9570-740ac953fa05" containerName="revoke-keys"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247509    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" containerName="revoke-keys"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247521    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247535    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247549    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="90006c25-90cb-49b2-9570-740ac953fa05" containerName="revoke-keys"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247558    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d52c78-f846-4674-9556-28940d3e6c13" containerName="istio-proxy"
Oct 29 23:28:50.250790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.247568    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy"
Oct 29 23:28:50.258868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.258202    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.276495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.275559    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:28:50.276495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.275643    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:28:50.276885 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.276525    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"]
Oct 29 23:28:50.353185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.353130    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.353388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.353352    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jk4d\" (UniqueName: \"kubernetes.io/projected/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-kube-api-access-4jk4d\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.353509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.353490    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.477784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.466426    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.496790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.491700    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jk4d\" (UniqueName: \"kubernetes.io/projected/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-kube-api-access-4jk4d\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.496790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.492034    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.500784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.498850    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.556234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.552521    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jk4d\" (UniqueName: \"kubernetes.io/projected/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-kube-api-access-4jk4d\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.574314 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.574283    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.596055 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.596017    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:50.916229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:50.916198    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"]
Oct 29 23:28:50.917436 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:28:50.917408    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98071347_8dd5_4bf6_84f1_8d04a4ae5cdf.slice/crio-25fbf9549006472df5753dec2ab2f25bf7bdadb8c854352d98fd1ecb5783fc35 WatchSource:0}: Error finding container 25fbf9549006472df5753dec2ab2f25bf7bdadb8c854352d98fd1ecb5783fc35: Status 404 returned error can't find the container with id 25fbf9549006472df5753dec2ab2f25bf7bdadb8c854352d98fd1ecb5783fc35
Oct 29 23:28:51.305001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:51.304970    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7" event={"ID":"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf","Type":"ContainerStarted","Data":"10af68e42af0ae70706b176b664d870863326853c96096a096198cfb6ae51e16"}
Oct 29 23:28:51.305148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:51.305019    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7" event={"ID":"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf","Type":"ContainerStarted","Data":"25fbf9549006472df5753dec2ab2f25bf7bdadb8c854352d98fd1ecb5783fc35"}
Oct 29 23:28:51.342478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:51.342450    4827 scope.go:122] "RemoveContainer" containerID="b88e3ad478fe8a50b38795e5973c45130d901abfbf1f392e1aabdfc6ae751180"
Oct 29 23:28:52.314831 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:52.314793    4827 generic.go:356] "Generic (PLEG): container finished" podID="98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" containerID="10af68e42af0ae70706b176b664d870863326853c96096a096198cfb6ae51e16" exitCode=0
Oct 29 23:28:52.315248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:52.314881    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7" event={"ID":"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf","Type":"ContainerDied","Data":"10af68e42af0ae70706b176b664d870863326853c96096a096198cfb6ae51e16"}
Oct 29 23:28:53.094036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.093977    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" podUID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" containerName="payload-processing" containerID="cri-o://e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8" gracePeriod=25
Oct 29 23:28:53.481232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.481206    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:53.568586 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.568548    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-xks-ca\") pod \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") "
Oct 29 23:28:53.568808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.568672    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-openshift-ca\") pod \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") "
Oct 29 23:28:53.568894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.568854    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-kube-api-access-4jk4d\" (UniqueName: \"kubernetes.io/projected/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-kube-api-access-4jk4d\") pod \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\" (UID: \"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf\") "
Oct 29 23:28:53.569084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.569058    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-openshift-ca" pod "98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" (UID: "98071347-8dd5-4bf6-84f1-8d04a4ae5cdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:53.569457 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.569436    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:53.570744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.570723    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-xks-ca" pod "98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" (UID: "98071347-8dd5-4bf6-84f1-8d04a4ae5cdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:53.571002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.570980    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-kube-api-access-4jk4d" pod "98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" (UID: "98071347-8dd5-4bf6-84f1-8d04a4ae5cdf"). InnerVolumeSpecName "kube-api-access-4jk4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:53.671507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.671426    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jk4d\" (UniqueName: \"kubernetes.io/projected/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-kube-api-access-4jk4d\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:53.671507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:53.671460    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.341555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.341521    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7" event={"ID":"98071347-8dd5-4bf6-84f1-8d04a4ae5cdf","Type":"ContainerDied","Data":"25fbf9549006472df5753dec2ab2f25bf7bdadb8c854352d98fd1ecb5783fc35"}
Oct 29 23:28:54.341716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.341607    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"
Oct 29 23:28:54.341716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.341609    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25fbf9549006472df5753dec2ab2f25bf7bdadb8c854352d98fd1ecb5783fc35"
Oct 29 23:28:54.694899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.694874    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:28:54.785739 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.785699    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-kube-api-access-l7j2s\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-kube-api-access-l7j2s\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.785935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.785753    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-credential-socket\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.785935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.785817    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-envoy\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.785935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.785864    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-data\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.785935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.785910    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-socket\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.786191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.786082    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-token\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-token\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.786191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.786113    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-podinfo\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.786191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.786157    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istiod-ca-cert\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.786353 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.786212    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-certs\") pod \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\" (UID: \"c38d50dd-ecfc-48ec-9dee-ea03ac789908\") "
Oct 29 23:28:54.786844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.786443    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-credential-socket" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:54.786844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.786730    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istiod-ca-cert" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:54.787125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.786913    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-certs" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:54.787125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.787013    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.787125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.787073    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.787125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.787105    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-data" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:54.787372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.787218    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-socket" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:54.788960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.788921    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-envoy" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:54.789092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.789050    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-kube-api-access-l7j2s" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "kube-api-access-l7j2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:54.789313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.789282    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-podinfo" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:28:54.789400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.789358    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-token" pod "c38d50dd-ecfc-48ec-9dee-ea03ac789908" (UID: "c38d50dd-ecfc-48ec-9dee-ea03ac789908"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:54.888593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.888565    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.888593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.888590    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.888806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.888600    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.888806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.888610    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7j2s\" (UniqueName: \"kubernetes.io/projected/c38d50dd-ecfc-48ec-9dee-ea03ac789908-kube-api-access-l7j2s\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.888806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.888620    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.888806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.888629    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.888806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.888636    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38d50dd-ecfc-48ec-9dee-ea03ac789908-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:54.910708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.910677    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"]
Oct 29 23:28:54.976143 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976066    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"]
Oct 29 23:28:54.976620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976607    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy"
Oct 29 23:28:54.976665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976621    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy"
Oct 29 23:28:54.976665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976638    4827 cpu_manager.go:414] "containerMap: removing container" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy"
Oct 29 23:28:54.976665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976643    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79663068-a8f3-4945-bcc6-fba2936cd3c5" containerName="istio-proxy"
Oct 29 23:28:54.976806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976675    4827 cpu_manager.go:414] "containerMap: removing container" podUID="98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" containerName="revoke-keys"
Oct 29 23:28:54.976806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976679    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" containerName="revoke-keys"
Oct 29 23:28:54.976806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976760    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" containerName="revoke-keys"
Oct 29 23:28:54.976806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976783    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy"
Oct 29 23:28:54.976806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.976792    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy"
Oct 29 23:28:54.980616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.980599    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:54.997188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.997154    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:28:54.997304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.997170    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:28:54.998884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.998853    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"]
Oct 29 23:28:54.999646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.999626    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy"
Oct 29 23:28:54.999646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:54.999644    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerName="istio-proxy"
Oct 29 23:28:55.003573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.003554    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"]
Oct 29 23:28:55.003694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.003667    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.024273 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.024235    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"]
Oct 29 23:28:55.090363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.090329    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.090516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.090446    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.090581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.090556    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dhg\" (UniqueName: \"kubernetes.io/projected/91e14136-c1d5-4f17-902c-f28aa8b2a182-kube-api-access-v8dhg\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.090625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.090585    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.090669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.090657    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.090716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.090681    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lgxq\" (UniqueName: \"kubernetes.io/projected/fcefe833-486a-465f-8303-8ab70934cd96-kube-api-access-2lgxq\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.191558 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.191523    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.191740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.191570    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lgxq\" (UniqueName: \"kubernetes.io/projected/fcefe833-486a-465f-8303-8ab70934cd96-kube-api-access-2lgxq\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.191819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.191739    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.191894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.191851    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.191966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.191950    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dhg\" (UniqueName: \"kubernetes.io/projected/91e14136-c1d5-4f17-902c-f28aa8b2a182-kube-api-access-v8dhg\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.192023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.191983    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.192328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.192302    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.192620 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.192599    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.194419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.194393    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.194509 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.194484    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.213555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.213530    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lgxq\" (UniqueName: \"kubernetes.io/projected/fcefe833-486a-465f-8303-8ab70934cd96-kube-api-access-2lgxq\") pod \"maas-api-revoke-sub-76ffe4d3-7nzzb\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.213978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.213958    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dhg\" (UniqueName: \"kubernetes.io/projected/91e14136-c1d5-4f17-902c-f28aa8b2a182-kube-api-access-v8dhg\") pod \"maas-api-revoke-sub-d66056f9-tv27j\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.295718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.295639    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:55.320024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.319995    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:55.371174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.369885    4827 generic.go:356] "Generic (PLEG): container finished" podID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" containerID="ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4" exitCode=0
Oct 29 23:28:55.372453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.371837    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"
Oct 29 23:28:55.373222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.372673    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" event={"ID":"c38d50dd-ecfc-48ec-9dee-ea03ac789908","Type":"ContainerDied","Data":"ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4"}
Oct 29 23:28:55.373222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.372738    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn" event={"ID":"c38d50dd-ecfc-48ec-9dee-ea03ac789908","Type":"ContainerDied","Data":"b49d611ba217557b11098f5fb934ca5a1f419e3a085899cebe2657cfe9ccf67c"}
Oct 29 23:28:55.373222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.372795    4827 scope.go:122] "RemoveContainer" containerID="ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4"
Oct 29 23:28:55.447939 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.447918    4827 scope.go:122] "RemoveContainer" containerID="50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417"
Oct 29 23:28:55.464840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.464818    4827 scope.go:122] "RemoveContainer" containerID="ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4"
Oct 29 23:28:55.465696 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:55.465194    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4\": container with ID starting with ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4 not found: ID does not exist" containerID="ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4"
Oct 29 23:28:55.465696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.465233    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4"} err="failed to get container status \"ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4\": rpc error: code = NotFound desc = could not find container \"ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4\": container with ID starting with ecc4a25e597b87c0762a22b8c79ef4fd4efd60fecd856504850433ea57daf7a4 not found: ID does not exist"
Oct 29 23:28:55.465696 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.465262    4827 scope.go:122] "RemoveContainer" containerID="50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417"
Oct 29 23:28:55.465989 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:55.465752    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417\": container with ID starting with 50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417 not found: ID does not exist" containerID="50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417"
Oct 29 23:28:55.465989 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.465801    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417"} err="failed to get container status \"50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417\": rpc error: code = NotFound desc = could not find container \"50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417\": container with ID starting with 50693603066af22858968f3882e323a72f709ba6002faef2b488d0f6ba98c417 not found: ID does not exist"
Oct 29 23:28:55.468872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.468815    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"]
Oct 29 23:28:55.473846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.473821    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5e52e0-openshift-default-5fc7fb5d4c-bdkgn"]
Oct 29 23:28:55.632580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.632557    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"]
Oct 29 23:28:55.633943 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:28:55.633914    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcefe833_486a_465f_8303_8ab70934cd96.slice/crio-d641eaf5ddf8d01b09d4021a6c07a74bb2cee5395b09b818298fc7e39ef55cad WatchSource:0}: Error finding container d641eaf5ddf8d01b09d4021a6c07a74bb2cee5395b09b818298fc7e39ef55cad: Status 404 returned error can't find the container with id d641eaf5ddf8d01b09d4021a6c07a74bb2cee5395b09b818298fc7e39ef55cad
Oct 29 23:28:55.689962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:55.689940    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"]
Oct 29 23:28:55.690910 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:28:55.690888    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e14136_c1d5_4f17_902c_f28aa8b2a182.slice/crio-af6544c2c28e74bc67a6c445975cc695b3cdc3ce9c38d20aeba7ccbf7e6c433a WatchSource:0}: Error finding container af6544c2c28e74bc67a6c445975cc695b3cdc3ce9c38d20aeba7ccbf7e6c433a: Status 404 returned error can't find the container with id af6544c2c28e74bc67a6c445975cc695b3cdc3ce9c38d20aeba7ccbf7e6c433a
Oct 29 23:28:56.381147 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:56.381112    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb" event={"ID":"fcefe833-486a-465f-8303-8ab70934cd96","Type":"ContainerStarted","Data":"3e89949ced5779c4677dd7151f265f845dbf13549ab1092165ea25c68020a6f1"}
Oct 29 23:28:56.381610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:56.381198    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb" event={"ID":"fcefe833-486a-465f-8303-8ab70934cd96","Type":"ContainerStarted","Data":"d641eaf5ddf8d01b09d4021a6c07a74bb2cee5395b09b818298fc7e39ef55cad"}
Oct 29 23:28:56.382665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:56.382639    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j" event={"ID":"91e14136-c1d5-4f17-902c-f28aa8b2a182","Type":"ContainerStarted","Data":"661eab3cb1c1ce1b11aa531efac8a8a49a22d49bfb829b2cd67188c7ca8b0047"}
Oct 29 23:28:56.382806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:56.382684    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j" event={"ID":"91e14136-c1d5-4f17-902c-f28aa8b2a182","Type":"ContainerStarted","Data":"af6544c2c28e74bc67a6c445975cc695b3cdc3ce9c38d20aeba7ccbf7e6c433a"}
Oct 29 23:28:56.421640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:56.421585    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb" podStartSLOduration=2.421573045 podStartE2EDuration="2.421573045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:56.417822397 +0000 UTC m=+2772.040733414" watchObservedRunningTime="2026-10-29 23:28:56.421573045 +0000 UTC m=+2772.044484070"
Oct 29 23:28:56.457808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:56.457743    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j" podStartSLOduration=2.457730531 podStartE2EDuration="2.457730531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:56.4550595 +0000 UTC m=+2772.077970520" watchObservedRunningTime="2026-10-29 23:28:56.457730531 +0000 UTC m=+2772.080641552"
Oct 29 23:28:56.489064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:56.489034    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38d50dd-ecfc-48ec-9dee-ea03ac789908" path="/var/lib/kubelet/pods/c38d50dd-ecfc-48ec-9dee-ea03ac789908/volumes"
Oct 29 23:28:57.391235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.391202    4827 generic.go:356] "Generic (PLEG): container finished" podID="fcefe833-486a-465f-8303-8ab70934cd96" containerID="3e89949ced5779c4677dd7151f265f845dbf13549ab1092165ea25c68020a6f1" exitCode=0
Oct 29 23:28:57.391723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.391299    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb" event={"ID":"fcefe833-486a-465f-8303-8ab70934cd96","Type":"ContainerDied","Data":"3e89949ced5779c4677dd7151f265f845dbf13549ab1092165ea25c68020a6f1"}
Oct 29 23:28:57.393006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.392973    4827 generic.go:356] "Generic (PLEG): container finished" podID="91e14136-c1d5-4f17-902c-f28aa8b2a182" containerID="661eab3cb1c1ce1b11aa531efac8a8a49a22d49bfb829b2cd67188c7ca8b0047" exitCode=0
Oct 29 23:28:57.393124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.393048    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j" event={"ID":"91e14136-c1d5-4f17-902c-f28aa8b2a182","Type":"ContainerDied","Data":"661eab3cb1c1ce1b11aa531efac8a8a49a22d49bfb829b2cd67188c7ca8b0047"}
Oct 29 23:28:57.435531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.435495    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" containerID="cri-o://72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774" gracePeriod=45
Oct 29 23:28:57.706693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.706669    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:28:57.750149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.750116    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" podUID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:28:57.846434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.846404    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e711f2b-f138-4812-adc1-07e9dd15aadd-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e711f2b-f138-4812-adc1-07e9dd15aadd-tls-certs\") pod \"2e711f2b-f138-4812-adc1-07e9dd15aadd\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") "
Oct 29 23:28:57.846634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.846489    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-tmp-dir\") pod \"2e711f2b-f138-4812-adc1-07e9dd15aadd\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") "
Oct 29 23:28:57.846712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.846645    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e711f2b-f138-4812-adc1-07e9dd15aadd-kube-api-access-75t4n\" (UniqueName: \"kubernetes.io/projected/2e711f2b-f138-4812-adc1-07e9dd15aadd-kube-api-access-75t4n\") pod \"2e711f2b-f138-4812-adc1-07e9dd15aadd\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") "
Oct 29 23:28:57.846712 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.846688    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-dshm\") pod \"2e711f2b-f138-4812-adc1-07e9dd15aadd\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") "
Oct 29 23:28:57.846853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.846793    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-home\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-home\") pod \"2e711f2b-f138-4812-adc1-07e9dd15aadd\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") "
Oct 29 23:28:57.846924 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.846904    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-model-cache\") pod \"2e711f2b-f138-4812-adc1-07e9dd15aadd\" (UID: \"2e711f2b-f138-4812-adc1-07e9dd15aadd\") "
Oct 29 23:28:57.847226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.847199    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-home" pod "2e711f2b-f138-4812-adc1-07e9dd15aadd" (UID: "2e711f2b-f138-4812-adc1-07e9dd15aadd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:57.847485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.847459    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-model-cache" pod "2e711f2b-f138-4812-adc1-07e9dd15aadd" (UID: "2e711f2b-f138-4812-adc1-07e9dd15aadd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:57.847622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.847585    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:57.847622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.847599    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:57.847760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.847642    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-tmp-dir" pod "2e711f2b-f138-4812-adc1-07e9dd15aadd" (UID: "2e711f2b-f138-4812-adc1-07e9dd15aadd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:57.849891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.849862    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e711f2b-f138-4812-adc1-07e9dd15aadd-tls-certs" pod "2e711f2b-f138-4812-adc1-07e9dd15aadd" (UID: "2e711f2b-f138-4812-adc1-07e9dd15aadd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:57.850034 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.849904    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e711f2b-f138-4812-adc1-07e9dd15aadd-kube-api-access-75t4n" pod "2e711f2b-f138-4812-adc1-07e9dd15aadd" (UID: "2e711f2b-f138-4812-adc1-07e9dd15aadd"). InnerVolumeSpecName "kube-api-access-75t4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:57.851585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.851547    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-dshm" pod "2e711f2b-f138-4812-adc1-07e9dd15aadd" (UID: "2e711f2b-f138-4812-adc1-07e9dd15aadd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:57.949348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.949267    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:57.949348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.949298    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75t4n\" (UniqueName: \"kubernetes.io/projected/2e711f2b-f138-4812-adc1-07e9dd15aadd-kube-api-access-75t4n\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:57.949348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.949307    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e711f2b-f138-4812-adc1-07e9dd15aadd-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:57.949348 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:57.949315    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e711f2b-f138-4812-adc1-07e9dd15aadd-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:58.401227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.401196    4827 generic.go:356] "Generic (PLEG): container finished" podID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerID="72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774" exitCode=0
Oct 29 23:28:58.401694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.401239    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"
Oct 29 23:28:58.401694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.401288    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" event={"ID":"2e711f2b-f138-4812-adc1-07e9dd15aadd","Type":"ContainerDied","Data":"72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774"}
Oct 29 23:28:58.401694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.401338    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" event={"ID":"2e711f2b-f138-4812-adc1-07e9dd15aadd","Type":"ContainerDied","Data":"8470b1edb5044b3c92d348e156e6be28a6ccd99fa5da23f07b5bfdb612c5a2fc"}
Oct 29 23:28:58.401694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.401360    4827 scope.go:122] "RemoveContainer" containerID="72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774"
Oct 29 23:28:58.414080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.414054    4827 scope.go:122] "RemoveContainer" containerID="72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774"
Oct 29 23:28:58.414409 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:28:58.414387    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774\": container with ID starting with 72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774 not found: ID does not exist" containerID="72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774"
Oct 29 23:28:58.414510 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.414418    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774"} err="failed to get container status \"72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774\": rpc error: code = NotFound desc = could not find container \"72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774\": container with ID starting with 72ac577e7bd5990e870fd3d9d48164ea4718b38bdb81e80dea9ee4eb9eeef774 not found: ID does not exist"
Oct 29 23:28:58.450463 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.450021    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"]
Oct 29 23:28:58.464015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.463682    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg"]
Oct 29 23:28:58.490069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.490036    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" path="/var/lib/kubelet/pods/2e711f2b-f138-4812-adc1-07e9dd15aadd/volumes"
Oct 29 23:28:58.619128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.619104    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:58.621386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.621362    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:28:58.634072 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.634045    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/premium-w3-17394a-kserve-855c947754-2hhkg" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:28:58.695845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.695664    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-openshift-ca\") pod \"91e14136-c1d5-4f17-902c-f28aa8b2a182\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") "
Oct 29 23:28:58.695845 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.695800    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-xks-ca\") pod \"91e14136-c1d5-4f17-902c-f28aa8b2a182\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") "
Oct 29 23:28:58.696084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.695856    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-openshift-ca\") pod \"fcefe833-486a-465f-8303-8ab70934cd96\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") "
Oct 29 23:28:58.696084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.695963    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-xks-ca\") pod \"fcefe833-486a-465f-8303-8ab70934cd96\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") "
Oct 29 23:28:58.696084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.696041    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcefe833-486a-465f-8303-8ab70934cd96-kube-api-access-2lgxq\" (UniqueName: \"kubernetes.io/projected/fcefe833-486a-465f-8303-8ab70934cd96-kube-api-access-2lgxq\") pod \"fcefe833-486a-465f-8303-8ab70934cd96\" (UID: \"fcefe833-486a-465f-8303-8ab70934cd96\") "
Oct 29 23:28:58.696249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.696091    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e14136-c1d5-4f17-902c-f28aa8b2a182-kube-api-access-v8dhg\" (UniqueName: \"kubernetes.io/projected/91e14136-c1d5-4f17-902c-f28aa8b2a182-kube-api-access-v8dhg\") pod \"91e14136-c1d5-4f17-902c-f28aa8b2a182\" (UID: \"91e14136-c1d5-4f17-902c-f28aa8b2a182\") "
Oct 29 23:28:58.706272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.704371    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-openshift-ca" pod "91e14136-c1d5-4f17-902c-f28aa8b2a182" (UID: "91e14136-c1d5-4f17-902c-f28aa8b2a182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:58.706272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.704858    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-openshift-ca" pod "fcefe833-486a-465f-8303-8ab70934cd96" (UID: "fcefe833-486a-465f-8303-8ab70934cd96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:58.712640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.712611    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e14136-c1d5-4f17-902c-f28aa8b2a182-kube-api-access-v8dhg" pod "91e14136-c1d5-4f17-902c-f28aa8b2a182" (UID: "91e14136-c1d5-4f17-902c-f28aa8b2a182"). InnerVolumeSpecName "kube-api-access-v8dhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:58.717849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.714912    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcefe833-486a-465f-8303-8ab70934cd96-kube-api-access-2lgxq" pod "fcefe833-486a-465f-8303-8ab70934cd96" (UID: "fcefe833-486a-465f-8303-8ab70934cd96"). InnerVolumeSpecName "kube-api-access-2lgxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:58.729839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.723836    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-xks-ca" pod "fcefe833-486a-465f-8303-8ab70934cd96" (UID: "fcefe833-486a-465f-8303-8ab70934cd96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:58.729839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.726992    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-xks-ca" pod "91e14136-c1d5-4f17-902c-f28aa8b2a182" (UID: "91e14136-c1d5-4f17-902c-f28aa8b2a182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:28:58.797914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.797884    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:58.797914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.797913    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91e14136-c1d5-4f17-902c-f28aa8b2a182-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:58.797914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.797923    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:58.798145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.797934    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcefe833-486a-465f-8303-8ab70934cd96-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:58.798145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.797942    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lgxq\" (UniqueName: \"kubernetes.io/projected/fcefe833-486a-465f-8303-8ab70934cd96-kube-api-access-2lgxq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:58.798145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:58.797950    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dhg\" (UniqueName: \"kubernetes.io/projected/91e14136-c1d5-4f17-902c-f28aa8b2a182-kube-api-access-v8dhg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:28:59.412870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:59.412831    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb" event={"ID":"fcefe833-486a-465f-8303-8ab70934cd96","Type":"ContainerDied","Data":"d641eaf5ddf8d01b09d4021a6c07a74bb2cee5395b09b818298fc7e39ef55cad"}
Oct 29 23:28:59.412870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:59.412869    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"
Oct 29 23:28:59.413597 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:59.412896    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d641eaf5ddf8d01b09d4021a6c07a74bb2cee5395b09b818298fc7e39ef55cad"
Oct 29 23:28:59.415797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:59.415732    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j" event={"ID":"91e14136-c1d5-4f17-902c-f28aa8b2a182","Type":"ContainerDied","Data":"af6544c2c28e74bc67a6c445975cc695b3cdc3ce9c38d20aeba7ccbf7e6c433a"}
Oct 29 23:28:59.415797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:59.415798    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6544c2c28e74bc67a6c445975cc695b3cdc3ce9c38d20aeba7ccbf7e6c433a"
Oct 29 23:28:59.416018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:28:59.415857    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"
Oct 29 23:29:00.729850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:00.729818    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"]
Oct 29 23:29:00.738288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:00.738257    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"]
Oct 29 23:29:02.824987 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:02.824955    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:29:05.735912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:05.735846    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc" podUID="2c0abba1-df13-4539-98f7-1c32afacef23" containerName="payload-processing" containerID="cri-o://6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2" gracePeriod=25
Oct 29 23:29:05.738793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:05.738745    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" podUID="a9db7699-245e-4237-9301-096107be2fa2" containerName="payload-pre-processing" containerID="cri-o://d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c" gracePeriod=25
Oct 29 23:29:05.999716 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:05.999651    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 23:29:06.025594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.025564    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0abba1-df13-4539-98f7-1c32afacef23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0abba1-df13-4539-98f7-1c32afacef23-plugins-config-volume\") pod \"2c0abba1-df13-4539-98f7-1c32afacef23\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") "
Oct 29 23:29:06.025594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.025600    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0abba1-df13-4539-98f7-1c32afacef23-kube-api-access-kk7zv\" (UniqueName: \"kubernetes.io/projected/2c0abba1-df13-4539-98f7-1c32afacef23-kube-api-access-kk7zv\") pod \"2c0abba1-df13-4539-98f7-1c32afacef23\" (UID: \"2c0abba1-df13-4539-98f7-1c32afacef23\") "
Oct 29 23:29:06.026051 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.026022    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0abba1-df13-4539-98f7-1c32afacef23-plugins-config-volume" pod "2c0abba1-df13-4539-98f7-1c32afacef23" (UID: "2c0abba1-df13-4539-98f7-1c32afacef23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:06.026258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.026242    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0abba1-df13-4539-98f7-1c32afacef23-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:06.027818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.027798    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0abba1-df13-4539-98f7-1c32afacef23-kube-api-access-kk7zv" pod "2c0abba1-df13-4539-98f7-1c32afacef23" (UID: "2c0abba1-df13-4539-98f7-1c32afacef23"). InnerVolumeSpecName "kube-api-access-kk7zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:06.127816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.127782    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk7zv\" (UniqueName: \"kubernetes.io/projected/2c0abba1-df13-4539-98f7-1c32afacef23-kube-api-access-kk7zv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:06.487469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.487436    4827 generic.go:356] "Generic (PLEG): container finished" podID="2c0abba1-df13-4539-98f7-1c32afacef23" containerID="6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2" exitCode=0
Oct 29 23:29:06.487729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.487559    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"
Oct 29 23:29:06.489278 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.489250    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc" event={"ID":"2c0abba1-df13-4539-98f7-1c32afacef23","Type":"ContainerDied","Data":"6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2"}
Oct 29 23:29:06.489409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.489295    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc" event={"ID":"2c0abba1-df13-4539-98f7-1c32afacef23","Type":"ContainerDied","Data":"386c5bbaeda9c56894aeeb46a5953a062d482ca4d1386da883f9f46e57fb0a16"}
Oct 29 23:29:06.489409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.489321    4827 scope.go:122] "RemoveContainer" containerID="6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2"
Oct 29 23:29:06.499895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.499877    4827 scope.go:122] "RemoveContainer" containerID="6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2"
Oct 29 23:29:06.500176 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:06.500160    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2\": container with ID starting with 6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2 not found: ID does not exist" containerID="6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2"
Oct 29 23:29:06.500229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.500183    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2"} err="failed to get container status \"6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2\": rpc error: code = NotFound desc = could not find container \"6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2\": container with ID starting with 6e13c494d6a2285938af15dac5c165dec71206e05854f68089542004af2585c2 not found: ID does not exist"
Oct 29 23:29:06.535412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.535387    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"]
Oct 29 23:29:06.545613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:06.545589    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b2921b-5cf46459b8-sz8lc"]
Oct 29 23:29:07.750232 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:07.750195    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" podUID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:29:08.489804 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:08.489756    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0abba1-df13-4539-98f7-1c32afacef23" path="/var/lib/kubelet/pods/2c0abba1-df13-4539-98f7-1c32afacef23/volumes"
Oct 29 23:29:09.933307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.933248    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main" containerID="cri-o://52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37" gracePeriod=45
Oct 29 23:29:09.966094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966069    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"]
Oct 29 23:29:09.966736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966719    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fcefe833-486a-465f-8303-8ab70934cd96" containerName="revoke-keys"
Oct 29 23:29:09.966803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966737    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcefe833-486a-465f-8303-8ab70934cd96" containerName="revoke-keys"
Oct 29 23:29:09.966803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966754    4827 cpu_manager.go:414] "containerMap: removing container" podUID="91e14136-c1d5-4f17-902c-f28aa8b2a182" containerName="revoke-keys"
Oct 29 23:29:09.966803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966760    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e14136-c1d5-4f17-902c-f28aa8b2a182" containerName="revoke-keys"
Oct 29 23:29:09.966803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966794    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2c0abba1-df13-4539-98f7-1c32afacef23" containerName="payload-processing"
Oct 29 23:29:09.966803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966799    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0abba1-df13-4539-98f7-1c32afacef23" containerName="payload-processing"
Oct 29 23:29:09.966977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966812    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main"
Oct 29 23:29:09.966977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966817    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main"
Oct 29 23:29:09.966977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966919    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0abba1-df13-4539-98f7-1c32afacef23" containerName="payload-processing"
Oct 29 23:29:09.966977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966930    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e711f2b-f138-4812-adc1-07e9dd15aadd" containerName="main"
Oct 29 23:29:09.966977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966939    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcefe833-486a-465f-8303-8ab70934cd96" containerName="revoke-keys"
Oct 29 23:29:09.966977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.966946    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e14136-c1d5-4f17-902c-f28aa8b2a182" containerName="revoke-keys"
Oct 29 23:29:09.970439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.970421    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:09.987084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.987064    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:29:09.997929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:09.997906    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:29:10.002650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.002625    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"]
Oct 29 23:29:10.087359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.087320    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb99s\" (UniqueName: \"kubernetes.io/projected/8a72d117-9954-4fbd-99b2-2854cffb4cb6-kube-api-access-wb99s\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.087546 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.087489    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.087669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.087648    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.188662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.188627    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb99s\" (UniqueName: \"kubernetes.io/projected/8a72d117-9954-4fbd-99b2-2854cffb4cb6-kube-api-access-wb99s\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.188862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.188740    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.188953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.188891    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.189692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.189649    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.191846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.191825    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.211271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.211240    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb99s\" (UniqueName: \"kubernetes.io/projected/8a72d117-9954-4fbd-99b2-2854cffb4cb6-kube-api-access-wb99s\") pod \"maas-api-revoke-sub-fd19522a-v4v9q\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.233377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.233348    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:29:10.290511 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.290477    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-tmp-dir\") pod \"94b989aa-8e73-42af-954c-76fbd2148199\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") "
Oct 29 23:29:10.290717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.290544    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b989aa-8e73-42af-954c-76fbd2148199-kube-api-access-skmn9\" (UniqueName: \"kubernetes.io/projected/94b989aa-8e73-42af-954c-76fbd2148199-kube-api-access-skmn9\") pod \"94b989aa-8e73-42af-954c-76fbd2148199\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") "
Oct 29 23:29:10.290717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.290588    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-dshm\") pod \"94b989aa-8e73-42af-954c-76fbd2148199\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") "
Oct 29 23:29:10.290717 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.290619    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-home\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-home\") pod \"94b989aa-8e73-42af-954c-76fbd2148199\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") "
Oct 29 23:29:10.291132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.290610    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-tmp-dir" pod "94b989aa-8e73-42af-954c-76fbd2148199" (UID: "94b989aa-8e73-42af-954c-76fbd2148199"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:10.291240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.291035    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-home" pod "94b989aa-8e73-42af-954c-76fbd2148199" (UID: "94b989aa-8e73-42af-954c-76fbd2148199"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:10.291484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.291457    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b989aa-8e73-42af-954c-76fbd2148199-tls-certs\" (UniqueName: \"kubernetes.io/secret/94b989aa-8e73-42af-954c-76fbd2148199-tls-certs\") pod \"94b989aa-8e73-42af-954c-76fbd2148199\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") "
Oct 29 23:29:10.291802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.291753    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-model-cache\") pod \"94b989aa-8e73-42af-954c-76fbd2148199\" (UID: \"94b989aa-8e73-42af-954c-76fbd2148199\") "
Oct 29 23:29:10.292126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.292101    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-model-cache" pod "94b989aa-8e73-42af-954c-76fbd2148199" (UID: "94b989aa-8e73-42af-954c-76fbd2148199"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:10.292376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.292355    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:10.292458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.292382    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:10.292458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.292394    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:10.294033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.294007    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b989aa-8e73-42af-954c-76fbd2148199-kube-api-access-skmn9" pod "94b989aa-8e73-42af-954c-76fbd2148199" (UID: "94b989aa-8e73-42af-954c-76fbd2148199"). InnerVolumeSpecName "kube-api-access-skmn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:10.294301 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.294271    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-dshm" pod "94b989aa-8e73-42af-954c-76fbd2148199" (UID: "94b989aa-8e73-42af-954c-76fbd2148199"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:10.294408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.294355    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b989aa-8e73-42af-954c-76fbd2148199-tls-certs" pod "94b989aa-8e73-42af-954c-76fbd2148199" (UID: "94b989aa-8e73-42af-954c-76fbd2148199"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:10.328756 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.328725    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:10.394894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.394255    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skmn9\" (UniqueName: \"kubernetes.io/projected/94b989aa-8e73-42af-954c-76fbd2148199-kube-api-access-skmn9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:10.394894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.394283    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b989aa-8e73-42af-954c-76fbd2148199-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:10.394894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.394293    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b989aa-8e73-42af-954c-76fbd2148199-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:10.578441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.578406    4827 generic.go:356] "Generic (PLEG): container finished" podID="94b989aa-8e73-42af-954c-76fbd2148199" containerID="52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37" exitCode=0
Oct 29 23:29:10.578588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.578568    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" event={"ID":"94b989aa-8e73-42af-954c-76fbd2148199","Type":"ContainerDied","Data":"52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37"}
Oct 29 23:29:10.578643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.578614    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc" event={"ID":"94b989aa-8e73-42af-954c-76fbd2148199","Type":"ContainerDied","Data":"d560a512850800f30ac57b464797202ce4e6ddd032b0067b8ea9debbb125feb6"}
Oct 29 23:29:10.578643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.578635    4827 scope.go:122] "RemoveContainer" containerID="52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37"
Oct 29 23:29:10.578857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.578844    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"
Oct 29 23:29:10.589835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.589817    4827 scope.go:122] "RemoveContainer" containerID="52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37"
Oct 29 23:29:10.590104 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:10.590082    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37\": container with ID starting with 52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37 not found: ID does not exist" containerID="52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37"
Oct 29 23:29:10.590164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.590112    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37"} err="failed to get container status \"52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37\": rpc error: code = NotFound desc = could not find container \"52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37\": container with ID starting with 52f222219ce4e6b02700956a207ff0f7c13ad50ac51175dd5af9aeebf0d40b37 not found: ID does not exist"
Oct 29 23:29:10.627621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.627592    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"]
Oct 29 23:29:10.631109 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.631085    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/premium-w5-b2921b-kserve-d6fdb7bc7-9fdxc"]
Oct 29 23:29:10.671013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.670987    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"]
Oct 29 23:29:10.676163 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:29:10.676135    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a72d117_9954_4fbd_99b2_2854cffb4cb6.slice/crio-63e5378a60ad433317141c1ee96801ea2bf1a3b026f444076f168bc938556c39 WatchSource:0}: Error finding container 63e5378a60ad433317141c1ee96801ea2bf1a3b026f444076f168bc938556c39: Status 404 returned error can't find the container with id 63e5378a60ad433317141c1ee96801ea2bf1a3b026f444076f168bc938556c39
Oct 29 23:29:10.944700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:10.944666    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" podUID="a9db7699-245e-4237-9301-096107be2fa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:29:11.588621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:11.588591    4827 generic.go:356] "Generic (PLEG): container finished" podID="8a72d117-9954-4fbd-99b2-2854cffb4cb6" containerID="579058a127abbef18360a74967a7c1c821d216a1ef491561ab4069c6fba346d3" exitCode=0
Oct 29 23:29:11.588805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:11.588674    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q" event={"ID":"8a72d117-9954-4fbd-99b2-2854cffb4cb6","Type":"ContainerDied","Data":"579058a127abbef18360a74967a7c1c821d216a1ef491561ab4069c6fba346d3"}
Oct 29 23:29:11.588805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:11.588734    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q" event={"ID":"8a72d117-9954-4fbd-99b2-2854cffb4cb6","Type":"ContainerStarted","Data":"63e5378a60ad433317141c1ee96801ea2bf1a3b026f444076f168bc938556c39"}
Oct 29 23:29:12.488752 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.488719    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b989aa-8e73-42af-954c-76fbd2148199" path="/var/lib/kubelet/pods/94b989aa-8e73-42af-954c-76fbd2148199/volumes"
Oct 29 23:29:12.748505 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.748439    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:12.824125 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.824091    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a72d117-9954-4fbd-99b2-2854cffb4cb6-kube-api-access-wb99s\" (UniqueName: \"kubernetes.io/projected/8a72d117-9954-4fbd-99b2-2854cffb4cb6-kube-api-access-wb99s\") pod \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") "
Oct 29 23:29:12.824327 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.824173    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-openshift-ca\") pod \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") "
Oct 29 23:29:12.824402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.824333    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-xks-ca\") pod \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\" (UID: \"8a72d117-9954-4fbd-99b2-2854cffb4cb6\") "
Oct 29 23:29:12.824592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.824563    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-openshift-ca" pod "8a72d117-9954-4fbd-99b2-2854cffb4cb6" (UID: "8a72d117-9954-4fbd-99b2-2854cffb4cb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:12.824887 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.824868    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:12.826377 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.826350    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a72d117-9954-4fbd-99b2-2854cffb4cb6-kube-api-access-wb99s" pod "8a72d117-9954-4fbd-99b2-2854cffb4cb6" (UID: "8a72d117-9954-4fbd-99b2-2854cffb4cb6"). InnerVolumeSpecName "kube-api-access-wb99s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:12.826799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.826757    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-xks-ca" pod "8a72d117-9954-4fbd-99b2-2854cffb4cb6" (UID: "8a72d117-9954-4fbd-99b2-2854cffb4cb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:12.925941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.925904    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb99s\" (UniqueName: \"kubernetes.io/projected/8a72d117-9954-4fbd-99b2-2854cffb4cb6-kube-api-access-wb99s\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:12.925941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:12.925941    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a72d117-9954-4fbd-99b2-2854cffb4cb6-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:13.351180 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.351122    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 23:29:13.431895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.431866    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-plugins-config-volume\") pod \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") "
Oct 29 23:29:13.432048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.432006    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-kube-api-access-xmfdc\" (UniqueName: \"kubernetes.io/projected/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-kube-api-access-xmfdc\") pod \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\" (UID: \"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1\") "
Oct 29 23:29:13.432252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.432231    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-plugins-config-volume" pod "1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" (UID: "1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:13.432501 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.432486    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:13.434107 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.434084    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-kube-api-access-xmfdc" pod "1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" (UID: "1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1"). InnerVolumeSpecName "kube-api-access-xmfdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:13.554829 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.545980    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfdc\" (UniqueName: \"kubernetes.io/projected/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1-kube-api-access-xmfdc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:13.602658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.602574    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q" event={"ID":"8a72d117-9954-4fbd-99b2-2854cffb4cb6","Type":"ContainerDied","Data":"63e5378a60ad433317141c1ee96801ea2bf1a3b026f444076f168bc938556c39"}
Oct 29 23:29:13.602658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.602634    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e5378a60ad433317141c1ee96801ea2bf1a3b026f444076f168bc938556c39"
Oct 29 23:29:13.602658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.602591    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"
Oct 29 23:29:13.604197 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.604175    4827 generic.go:356] "Generic (PLEG): container finished" podID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" containerID="e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8" exitCode=0
Oct 29 23:29:13.604331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.604218    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"
Oct 29 23:29:13.604331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.604264    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" event={"ID":"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1","Type":"ContainerDied","Data":"e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8"}
Oct 29 23:29:13.604331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.604321    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb" event={"ID":"1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1","Type":"ContainerDied","Data":"933ab46da54106364bb04fc32d562cb94343b85bf9a5fb4641ad008da88f7f5b"}
Oct 29 23:29:13.604455 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.604353    4827 scope.go:122] "RemoveContainer" containerID="e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8"
Oct 29 23:29:13.615451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.615437    4827 scope.go:122] "RemoveContainer" containerID="e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8"
Oct 29 23:29:13.615733 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:13.615715    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8\": container with ID starting with e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8 not found: ID does not exist" containerID="e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8"
Oct 29 23:29:13.615799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.615740    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8"} err="failed to get container status \"e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8\": rpc error: code = NotFound desc = could not find container \"e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8\": container with ID starting with e8cbe3af633f1a006807844ae3fa9012136abe6c690c46954670d3bfb3fdd3d8 not found: ID does not exist"
Oct 29 23:29:13.654132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.654104    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"]
Oct 29 23:29:13.660425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:13.660401    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-17394a-56cfdb64f8-z9rbb"]
Oct 29 23:29:14.488012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:14.487974    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" path="/var/lib/kubelet/pods/1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1/volumes"
Oct 29 23:29:16.435282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.435248    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"]
Oct 29 23:29:16.435881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.435863    4827 cpu_manager.go:414] "containerMap: removing container" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main"
Oct 29 23:29:16.435881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.435880    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main"
Oct 29 23:29:16.436038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.435899    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8a72d117-9954-4fbd-99b2-2854cffb4cb6" containerName="revoke-keys"
Oct 29 23:29:16.436038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.435904    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72d117-9954-4fbd-99b2-2854cffb4cb6" containerName="revoke-keys"
Oct 29 23:29:16.436038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.435933    4827 cpu_manager.go:414] "containerMap: removing container" podUID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" containerName="payload-processing"
Oct 29 23:29:16.436038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.435938    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" containerName="payload-processing"
Oct 29 23:29:16.436038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.436017    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b989aa-8e73-42af-954c-76fbd2148199" containerName="main"
Oct 29 23:29:16.436038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.436027    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72d117-9954-4fbd-99b2-2854cffb4cb6" containerName="revoke-keys"
Oct 29 23:29:16.436038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.436038    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dde070a-d0fa-4dd9-9fd4-eb1affab6cb1" containerName="payload-processing"
Oct 29 23:29:16.440086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.440066    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.468011 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.467957    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"sub-test-model-af237d-kserve-self-signed-certs\""
Oct 29 23:29:16.468867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.468027    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"openshift-service-ca.crt\""
Oct 29 23:29:16.468867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.468400    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"kube-root-ca.crt\""
Oct 29 23:29:16.468867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.468562    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"default-dockercfg-smz99\""
Oct 29 23:29:16.472503 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.472475    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"]
Oct 29 23:29:16.584797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.584736    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tls-certs\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.585000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.584930    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tmp-dir\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.585000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.584982    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-dshm\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.585140 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.585121    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzpxg\" (UniqueName: \"kubernetes.io/projected/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-kube-api-access-kzpxg\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.585202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.585185    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-model-cache\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.585331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.585309    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-home\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.691693 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.691603    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-home\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.692007 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.691980    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tls-certs\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.692215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.692200    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tmp-dir\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.692358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.692334    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-dshm\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.692524 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.692507    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzpxg\" (UniqueName: \"kubernetes.io/projected/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-kube-api-access-kzpxg\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.699947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.699900    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-model-cache\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.700602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.698915    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tmp-dir\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.700602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.698646    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-home\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.700602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.700453    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-model-cache\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.702651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.702618    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tls-certs\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.737786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.732113    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-dshm\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.740585 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.740511    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzpxg\" (UniqueName: \"kubernetes.io/projected/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-kube-api-access-kzpxg\") pod \"sub-test-model-af237d-kserve-79d79c8995-bsfmf\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") " pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:16.760870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:16.755058    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:17.168497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:17.168470    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"]
Oct 29 23:29:17.168817 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:29:17.168792    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda38f74ea_d450_44f5_8c7c_7a110cbc34c7.slice/crio-91f83e6b5d2cd3eb23955f81a4bfd0d55c98203e3e3b0656be1f89feae82dcf6 WatchSource:0}: Error finding container 91f83e6b5d2cd3eb23955f81a4bfd0d55c98203e3e3b0656be1f89feae82dcf6: Status 404 returned error can't find the container with id 91f83e6b5d2cd3eb23955f81a4bfd0d55c98203e3e3b0656be1f89feae82dcf6
Oct 29 23:29:17.632796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:17.632746    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf" event={"ID":"a38f74ea-d450-44f5-8c7c-7a110cbc34c7","Type":"ContainerStarted","Data":"8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162"}
Oct 29 23:29:17.633197 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:17.632804    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf" event={"ID":"a38f74ea-d450-44f5-8c7c-7a110cbc34c7","Type":"ContainerStarted","Data":"91f83e6b5d2cd3eb23955f81a4bfd0d55c98203e3e3b0656be1f89feae82dcf6"}
Oct 29 23:29:17.672378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:17.672308    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf" podStartSLOduration=1.672292282 podStartE2EDuration="1.672292282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:17.670692399 +0000 UTC m=+2793.293603421" watchObservedRunningTime="2026-10-29 23:29:17.672292282 +0000 UTC m=+2793.295203305"
Oct 29 23:29:17.761160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:17.761122    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:18.197300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.197261    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"]
Oct 29 23:29:18.201339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.201312    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.214631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.214601    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:29:18.225300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.225277    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:29:18.230342 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.230313    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"]
Oct 29 23:29:18.335104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.335069    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.335276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.335127    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72gs\" (UniqueName: \"kubernetes.io/projected/4203bcb2-2d10-4d67-ae1d-c7764f116179-kube-api-access-r72gs\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.335326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.335296    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.436115 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.436082    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.436281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.436200    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.436281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.436235    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72gs\" (UniqueName: \"kubernetes.io/projected/4203bcb2-2d10-4d67-ae1d-c7764f116179-kube-api-access-r72gs\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.436906 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.436882    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.438486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.438459    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.457973 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.457916    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72gs\" (UniqueName: \"kubernetes.io/projected/4203bcb2-2d10-4d67-ae1d-c7764f116179-kube-api-access-r72gs\") pod \"maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.516179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.516151    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:18.817758 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:18.817731    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"]
Oct 29 23:29:18.818754 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:29:18.818727    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4203bcb2_2d10_4d67_ae1d_c7764f116179.slice/crio-9af1f651d4503f920e580587c2b5e3aebcabd5ae92b460c1002fdae6a1138266 WatchSource:0}: Error finding container 9af1f651d4503f920e580587c2b5e3aebcabd5ae92b460c1002fdae6a1138266: Status 404 returned error can't find the container with id 9af1f651d4503f920e580587c2b5e3aebcabd5ae92b460c1002fdae6a1138266
Oct 29 23:29:19.651193 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:19.651157    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx" event={"ID":"4203bcb2-2d10-4d67-ae1d-c7764f116179","Type":"ContainerStarted","Data":"5c72d50920119a133b7ec448336125a09f4148e1d47ecb34eb9189f313ca8a8a"}
Oct 29 23:29:19.651406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:19.651227    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx" event={"ID":"4203bcb2-2d10-4d67-ae1d-c7764f116179","Type":"ContainerStarted","Data":"9af1f651d4503f920e580587c2b5e3aebcabd5ae92b460c1002fdae6a1138266"}
Oct 29 23:29:19.691142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:19.691082    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx" podStartSLOduration=1.691062842 podStartE2EDuration="1.691062842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:19.689083175 +0000 UTC m=+2795.311994212" watchObservedRunningTime="2026-10-29 23:29:19.691062842 +0000 UTC m=+2795.313973862"
Oct 29 23:29:20.661894 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:20.661855    4827 generic.go:356] "Generic (PLEG): container finished" podID="4203bcb2-2d10-4d67-ae1d-c7764f116179" containerID="5c72d50920119a133b7ec448336125a09f4148e1d47ecb34eb9189f313ca8a8a" exitCode=0
Oct 29 23:29:20.662299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:20.661931    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx" event={"ID":"4203bcb2-2d10-4d67-ae1d-c7764f116179","Type":"ContainerDied","Data":"5c72d50920119a133b7ec448336125a09f4148e1d47ecb34eb9189f313ca8a8a"}
Oct 29 23:29:20.944168 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:20.944073    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" podUID="a9db7699-245e-4237-9301-096107be2fa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:29:21.643471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.643438    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"]
Oct 29 23:29:21.647777 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.647722    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.676321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.674200    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"]
Oct 29 23:29:21.676321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.676215    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.677036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.676328    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.677036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.676364    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jqm\" (UniqueName: \"kubernetes.io/projected/09b13b1d-e33d-4399-bc4d-175bdafcee1c-kube-api-access-h7jqm\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.777694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.777652    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.777920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.777788    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.777920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.777826    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jqm\" (UniqueName: \"kubernetes.io/projected/09b13b1d-e33d-4399-bc4d-175bdafcee1c-kube-api-access-h7jqm\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.779045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.779018    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.782081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.782054    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.801043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.801014    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jqm\" (UniqueName: \"kubernetes.io/projected/09b13b1d-e33d-4399-bc4d-175bdafcee1c-kube-api-access-h7jqm\") pod \"maas-api-revoke-sub-937d9a20-jb9lf\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:21.834337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.834316    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:21.878641 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.878608    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-openshift-ca\") pod \"4203bcb2-2d10-4d67-ae1d-c7764f116179\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") "
Oct 29 23:29:21.878847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.878653    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-xks-ca\") pod \"4203bcb2-2d10-4d67-ae1d-c7764f116179\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") "
Oct 29 23:29:21.878847 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.878711    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4203bcb2-2d10-4d67-ae1d-c7764f116179-kube-api-access-r72gs\" (UniqueName: \"kubernetes.io/projected/4203bcb2-2d10-4d67-ae1d-c7764f116179-kube-api-access-r72gs\") pod \"4203bcb2-2d10-4d67-ae1d-c7764f116179\" (UID: \"4203bcb2-2d10-4d67-ae1d-c7764f116179\") "
Oct 29 23:29:21.879041 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.879016    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-openshift-ca" pod "4203bcb2-2d10-4d67-ae1d-c7764f116179" (UID: "4203bcb2-2d10-4d67-ae1d-c7764f116179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:21.879395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.879373    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:21.881277 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.881243    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4203bcb2-2d10-4d67-ae1d-c7764f116179-kube-api-access-r72gs" pod "4203bcb2-2d10-4d67-ae1d-c7764f116179" (UID: "4203bcb2-2d10-4d67-ae1d-c7764f116179"). InnerVolumeSpecName "kube-api-access-r72gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:21.881425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.881404    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-xks-ca" pod "4203bcb2-2d10-4d67-ae1d-c7764f116179" (UID: "4203bcb2-2d10-4d67-ae1d-c7764f116179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:21.981255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.981176    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4203bcb2-2d10-4d67-ae1d-c7764f116179-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:21.981255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.981203    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72gs\" (UniqueName: \"kubernetes.io/projected/4203bcb2-2d10-4d67-ae1d-c7764f116179-kube-api-access-r72gs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:21.985688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:21.985661    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:22.308306 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:22.308277    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"]
Oct 29 23:29:22.308725 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:29:22.308701    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b13b1d_e33d_4399_bc4d_175bdafcee1c.slice/crio-e84ea2de73ff1da2ae87ebf15164c3602bd39484782a6f42140168f256bf267e WatchSource:0}: Error finding container e84ea2de73ff1da2ae87ebf15164c3602bd39484782a6f42140168f256bf267e: Status 404 returned error can't find the container with id e84ea2de73ff1da2ae87ebf15164c3602bd39484782a6f42140168f256bf267e
Oct 29 23:29:22.683595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:22.683555    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"
Oct 29 23:29:22.684057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:22.683599    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx" event={"ID":"4203bcb2-2d10-4d67-ae1d-c7764f116179","Type":"ContainerDied","Data":"9af1f651d4503f920e580587c2b5e3aebcabd5ae92b460c1002fdae6a1138266"}
Oct 29 23:29:22.684057 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:22.683649    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af1f651d4503f920e580587c2b5e3aebcabd5ae92b460c1002fdae6a1138266"
Oct 29 23:29:22.685435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:22.685391    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf" event={"ID":"09b13b1d-e33d-4399-bc4d-175bdafcee1c","Type":"ContainerStarted","Data":"43a6afcce2340b2ca40fd180ed848974440d4482c77361c5086b9f5f67afd4f5"}
Oct 29 23:29:22.685588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:22.685440    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf" event={"ID":"09b13b1d-e33d-4399-bc4d-175bdafcee1c","Type":"ContainerStarted","Data":"e84ea2de73ff1da2ae87ebf15164c3602bd39484782a6f42140168f256bf267e"}
Oct 29 23:29:22.722960 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:22.722914    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf" podStartSLOduration=1.722902792 podStartE2EDuration="1.722902792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:22.722105719 +0000 UTC m=+2798.345016738" watchObservedRunningTime="2026-10-29 23:29:22.722902792 +0000 UTC m=+2798.345813808"
Oct 29 23:29:23.811787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:23.809176    4827 generic.go:356] "Generic (PLEG): container finished" podID="09b13b1d-e33d-4399-bc4d-175bdafcee1c" containerID="43a6afcce2340b2ca40fd180ed848974440d4482c77361c5086b9f5f67afd4f5" exitCode=0
Oct 29 23:29:23.811787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:23.809372    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf" event={"ID":"09b13b1d-e33d-4399-bc4d-175bdafcee1c","Type":"ContainerDied","Data":"43a6afcce2340b2ca40fd180ed848974440d4482c77361c5086b9f5f67afd4f5"}
Oct 29 23:29:23.982513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:23.982475    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"]
Oct 29 23:29:23.985834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:23.985811    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17394a-vsxmn"]
Oct 29 23:29:24.497919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:24.497884    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90006c25-90cb-49b2-9570-740ac953fa05" path="/var/lib/kubelet/pods/90006c25-90cb-49b2-9570-740ac953fa05/volumes"
Oct 29 23:29:24.973857 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:24.973832    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:25.155610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.155565    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-xks-ca\") pod \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") "
Oct 29 23:29:25.155814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.155719    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b13b1d-e33d-4399-bc4d-175bdafcee1c-kube-api-access-h7jqm\" (UniqueName: \"kubernetes.io/projected/09b13b1d-e33d-4399-bc4d-175bdafcee1c-kube-api-access-h7jqm\") pod \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") "
Oct 29 23:29:25.155814 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.155744    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-openshift-ca\") pod \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\" (UID: \"09b13b1d-e33d-4399-bc4d-175bdafcee1c\") "
Oct 29 23:29:25.156386 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.156350    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-openshift-ca" pod "09b13b1d-e33d-4399-bc4d-175bdafcee1c" (UID: "09b13b1d-e33d-4399-bc4d-175bdafcee1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:25.158411 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.158382    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b13b1d-e33d-4399-bc4d-175bdafcee1c-kube-api-access-h7jqm" pod "09b13b1d-e33d-4399-bc4d-175bdafcee1c" (UID: "09b13b1d-e33d-4399-bc4d-175bdafcee1c"). InnerVolumeSpecName "kube-api-access-h7jqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:25.158545 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.158431    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-xks-ca" pod "09b13b1d-e33d-4399-bc4d-175bdafcee1c" (UID: "09b13b1d-e33d-4399-bc4d-175bdafcee1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:25.257987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.257955    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7jqm\" (UniqueName: \"kubernetes.io/projected/09b13b1d-e33d-4399-bc4d-175bdafcee1c-kube-api-access-h7jqm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:25.257987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.257981    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:25.257987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.257991    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b13b1d-e33d-4399-bc4d-175bdafcee1c-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:25.823463 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.823436    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"
Oct 29 23:29:25.823675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.823458    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf" event={"ID":"09b13b1d-e33d-4399-bc4d-175bdafcee1c","Type":"ContainerDied","Data":"e84ea2de73ff1da2ae87ebf15164c3602bd39484782a6f42140168f256bf267e"}
Oct 29 23:29:25.823675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:25.823512    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e84ea2de73ff1da2ae87ebf15164c3602bd39484782a6f42140168f256bf267e"
Oct 29 23:29:26.016198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.016140    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 23:29:26.171841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.171810    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9db7699-245e-4237-9301-096107be2fa2-kube-api-access-m4xn8\" (UniqueName: \"kubernetes.io/projected/a9db7699-245e-4237-9301-096107be2fa2-kube-api-access-m4xn8\") pod \"a9db7699-245e-4237-9301-096107be2fa2\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") "
Oct 29 23:29:26.172032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.171886    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9db7699-245e-4237-9301-096107be2fa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db7699-245e-4237-9301-096107be2fa2-plugins-config-volume\") pod \"a9db7699-245e-4237-9301-096107be2fa2\" (UID: \"a9db7699-245e-4237-9301-096107be2fa2\") "
Oct 29 23:29:26.172253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.172228    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9db7699-245e-4237-9301-096107be2fa2-plugins-config-volume" pod "a9db7699-245e-4237-9301-096107be2fa2" (UID: "a9db7699-245e-4237-9301-096107be2fa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:26.174218 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.174199    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9db7699-245e-4237-9301-096107be2fa2-kube-api-access-m4xn8" pod "a9db7699-245e-4237-9301-096107be2fa2" (UID: "a9db7699-245e-4237-9301-096107be2fa2"). InnerVolumeSpecName "kube-api-access-m4xn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:26.273662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.273598    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4xn8\" (UniqueName: \"kubernetes.io/projected/a9db7699-245e-4237-9301-096107be2fa2-kube-api-access-m4xn8\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:26.273662 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.273623    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db7699-245e-4237-9301-096107be2fa2-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:26.761335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.761301    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:26.774084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.774057    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:26.831169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.831141    4827 generic.go:356] "Generic (PLEG): container finished" podID="a9db7699-245e-4237-9301-096107be2fa2" containerID="d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c" exitCode=0
Oct 29 23:29:26.831361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.831193    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"
Oct 29 23:29:26.831361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.831220    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" event={"ID":"a9db7699-245e-4237-9301-096107be2fa2","Type":"ContainerDied","Data":"d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c"}
Oct 29 23:29:26.831361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.831268    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw" event={"ID":"a9db7699-245e-4237-9301-096107be2fa2","Type":"ContainerDied","Data":"a42083102132b9052bd127eb86f97ca1d0485c70da2002d3ca7a1e52aa5652a4"}
Oct 29 23:29:26.831361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.831290    4827 scope.go:122] "RemoveContainer" containerID="d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c"
Oct 29 23:29:26.842362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.842343    4827 scope.go:122] "RemoveContainer" containerID="d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c"
Oct 29 23:29:26.842635 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:26.842616    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c\": container with ID starting with d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c not found: ID does not exist" containerID="d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c"
Oct 29 23:29:26.842689 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.842641    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c"} err="failed to get container status \"d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c\": rpc error: code = NotFound desc = could not find container \"d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c\": container with ID starting with d073d0aefe5d5ced4f747e5e856bad096861eede74a52c31b96f9e1e2997e84c not found: ID does not exist"
Oct 29 23:29:26.844536 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.844516    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:29:26.923325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.923293    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"]
Oct 29 23:29:26.927968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:26.927942    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b2921b-79d8456c4c-v2mnw"]
Oct 29 23:29:28.496949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:28.496918    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9db7699-245e-4237-9301-096107be2fa2" path="/var/lib/kubelet/pods/a9db7699-245e-4237-9301-096107be2fa2/volumes"
Oct 29 23:29:32.156561 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:32.156525    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"]
Oct 29 23:29:32.157186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:32.156798    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy" containerID="cri-o://2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593" gracePeriod=30
Oct 29 23:29:32.834383 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:32.834349    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:29:37.429508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.429445    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 23:29:37.539281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539249    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-certs\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539299    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-podinfo\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539335    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-envoy\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539370    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6022bfd-dfbd-45c3-8209-5697f8517b36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6022bfd-dfbd-45c3-8209-5697f8517b36-istiod-ca-cert\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539403    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-token\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-token\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539483    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-socket\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539540    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-certs" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:37.539736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539580    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-data\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539682    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-credential-socket\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.539736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539722    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-kube-api-access-dtd9d\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-kube-api-access-dtd9d\") pod \"e6022bfd-dfbd-45c3-8209-5697f8517b36\" (UID: \"e6022bfd-dfbd-45c3-8209-5697f8517b36\") "
Oct 29 23:29:37.540073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539814    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-socket" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:37.540073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539873    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6022bfd-dfbd-45c3-8209-5697f8517b36-istiod-ca-cert" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:37.540073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.539914    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-data" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:37.540073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.540018    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-credential-socket" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:37.540323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.540299    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.540389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.540327    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.540389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.540341    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.540389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.540354    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.540389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.540370    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6022bfd-dfbd-45c3-8209-5697f8517b36-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.541534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.541508    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-podinfo" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:29:37.541688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.541668    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-token" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:37.541808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.541679    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-envoy" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:37.542037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.542017    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-kube-api-access-dtd9d" pod "e6022bfd-dfbd-45c3-8209-5697f8517b36" (UID: "e6022bfd-dfbd-45c3-8209-5697f8517b36"). InnerVolumeSpecName "kube-api-access-dtd9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:37.641742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.641707    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtd9d\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-kube-api-access-dtd9d\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.641742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.641733    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.641742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.641742    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.641742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.641751    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6022bfd-dfbd-45c3-8209-5697f8517b36-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:37.922227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.922195    4827 generic.go:356] "Generic (PLEG): container finished" podID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerID="2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593" exitCode=0
Oct 29 23:29:37.922402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.922240    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" event={"ID":"e6022bfd-dfbd-45c3-8209-5697f8517b36","Type":"ContainerDied","Data":"2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593"}
Oct 29 23:29:37.922402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.922249    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"
Oct 29 23:29:37.922402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.922284    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4" event={"ID":"e6022bfd-dfbd-45c3-8209-5697f8517b36","Type":"ContainerDied","Data":"bec8cdd2dfaeed11d462b70b8a9c2642813f85fc7d54bcc86d59c191098b068c"}
Oct 29 23:29:37.922402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.922305    4827 scope.go:122] "RemoveContainer" containerID="2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593"
Oct 29 23:29:37.933685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.933667    4827 scope.go:122] "RemoveContainer" containerID="84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a"
Oct 29 23:29:37.943632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.943614    4827 scope.go:122] "RemoveContainer" containerID="2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593"
Oct 29 23:29:37.944032 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:37.944013    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593\": container with ID starting with 2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593 not found: ID does not exist" containerID="2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593"
Oct 29 23:29:37.944102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.944038    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593"} err="failed to get container status \"2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593\": rpc error: code = NotFound desc = could not find container \"2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593\": container with ID starting with 2220a5d77c5e2f21281bae7fdf2662212615f1c3ecda8dac37f39e81ee022593 not found: ID does not exist"
Oct 29 23:29:37.944102 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.944057    4827 scope.go:122] "RemoveContainer" containerID="84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a"
Oct 29 23:29:37.944322 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:37.944303    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a\": container with ID starting with 84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a not found: ID does not exist" containerID="84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a"
Oct 29 23:29:37.944375 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.944325    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a"} err="failed to get container status \"84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a\": rpc error: code = NotFound desc = could not find container \"84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a\": container with ID starting with 84db3adc99bb97222d2eeb73b0251b90c6379035dac38123101fd862f6809d0a not found: ID does not exist"
Oct 29 23:29:37.987568 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.987541    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"]
Oct 29 23:29:37.998053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:37.998026    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-d8dv4"]
Oct 29 23:29:38.488624 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:38.488589    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" path="/var/lib/kubelet/pods/e6022bfd-dfbd-45c3-8209-5697f8517b36/volumes"
Oct 29 23:29:39.363324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:39.363290    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"]
Oct 29 23:29:39.373793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:39.373750    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-31226c-vltkv"]
Oct 29 23:29:40.568392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:40.568345    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5c492a-c8aa-444f-af7e-e906c4f7b4cc" path="/var/lib/kubelet/pods/fb5c492a-c8aa-444f-af7e-e906c4f7b4cc/volumes"
Oct 29 23:29:41.289379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.289345    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"]
Oct 29 23:29:41.289922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.289889    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs" podUID="e87138d4-fac8-4de7-b5d0-021f38532334" containerName="maas-api" containerID="cri-o://ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123" gracePeriod=30
Oct 29 23:29:41.848179 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.848152    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:29:41.959590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.959548    4827 generic.go:356] "Generic (PLEG): container finished" podID="e87138d4-fac8-4de7-b5d0-021f38532334" containerID="ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123" exitCode=0
Oct 29 23:29:41.959748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.959599    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"
Oct 29 23:29:41.959748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.959637    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs" event={"ID":"e87138d4-fac8-4de7-b5d0-021f38532334","Type":"ContainerDied","Data":"ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123"}
Oct 29 23:29:41.959748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.959696    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs" event={"ID":"e87138d4-fac8-4de7-b5d0-021f38532334","Type":"ContainerDied","Data":"213e6b8def3261b138c7b4a668dd2062c70353b14c34d70b0e9b7c19f5896227"}
Oct 29 23:29:41.959748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.959726    4827 scope.go:122] "RemoveContainer" containerID="ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123"
Oct 29 23:29:41.969614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.969593    4827 scope.go:122] "RemoveContainer" containerID="ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123"
Oct 29 23:29:41.969917 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:41.969897    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123\": container with ID starting with ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123 not found: ID does not exist" containerID="ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123"
Oct 29 23:29:41.970001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:41.969925    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123"} err="failed to get container status \"ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123\": rpc error: code = NotFound desc = could not find container \"ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123\": container with ID starting with ca3bc0f82645d7275367729f2968342ee02380011be27e27ae4b5f6f55620123 not found: ID does not exist"
Oct 29 23:29:42.011969 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.011934    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87138d4-fac8-4de7-b5d0-021f38532334-kube-api-access-sz29w\" (UniqueName: \"kubernetes.io/projected/e87138d4-fac8-4de7-b5d0-021f38532334-kube-api-access-sz29w\") pod \"e87138d4-fac8-4de7-b5d0-021f38532334\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") "
Oct 29 23:29:42.012185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.011989    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-maas-api-tls\") pod \"e87138d4-fac8-4de7-b5d0-021f38532334\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") "
Oct 29 23:29:42.012185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.012055    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87138d4-fac8-4de7-b5d0-021f38532334-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87138d4-fac8-4de7-b5d0-021f38532334-openshift-service-ca\") pod \"e87138d4-fac8-4de7-b5d0-021f38532334\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") "
Oct 29 23:29:42.012185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.012144    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-metrics-certs\") pod \"e87138d4-fac8-4de7-b5d0-021f38532334\" (UID: \"e87138d4-fac8-4de7-b5d0-021f38532334\") "
Oct 29 23:29:42.012483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.012448    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87138d4-fac8-4de7-b5d0-021f38532334-openshift-service-ca" pod "e87138d4-fac8-4de7-b5d0-021f38532334" (UID: "e87138d4-fac8-4de7-b5d0-021f38532334"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:42.012866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.012838    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87138d4-fac8-4de7-b5d0-021f38532334-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:42.014300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.014272    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-maas-api-tls" pod "e87138d4-fac8-4de7-b5d0-021f38532334" (UID: "e87138d4-fac8-4de7-b5d0-021f38532334"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:42.014300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.014286    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-metrics-certs" pod "e87138d4-fac8-4de7-b5d0-021f38532334" (UID: "e87138d4-fac8-4de7-b5d0-021f38532334"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:42.014446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.014332    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87138d4-fac8-4de7-b5d0-021f38532334-kube-api-access-sz29w" pod "e87138d4-fac8-4de7-b5d0-021f38532334" (UID: "e87138d4-fac8-4de7-b5d0-021f38532334"). InnerVolumeSpecName "kube-api-access-sz29w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:42.115236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.115141    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz29w\" (UniqueName: \"kubernetes.io/projected/e87138d4-fac8-4de7-b5d0-021f38532334-kube-api-access-sz29w\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:42.115236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.115182    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:42.115236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.115197    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87138d4-fac8-4de7-b5d0-021f38532334-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:42.130274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.130182    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"]
Oct 29 23:29:42.137221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.137193    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d24e75-cj5tx"]
Oct 29 23:29:42.344391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.344357    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"]
Oct 29 23:29:42.348473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.348447    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d24e75-9df44d66d-9n2vs"]
Oct 29 23:29:42.489099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.489065    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4203bcb2-2d10-4d67-ae1d-c7764f116179" path="/var/lib/kubelet/pods/4203bcb2-2d10-4d67-ae1d-c7764f116179/volumes"
Oct 29 23:29:42.489514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:42.489501    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87138d4-fac8-4de7-b5d0-021f38532334" path="/var/lib/kubelet/pods/e87138d4-fac8-4de7-b5d0-021f38532334/volumes"
Oct 29 23:29:45.872640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:45.872611    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"]
Oct 29 23:29:45.882817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:45.882791    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b2921b-nn8k7"]
Oct 29 23:29:46.488026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:46.487989    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98071347-8dd5-4bf6-84f1-8d04a4ae5cdf" path="/var/lib/kubelet/pods/98071347-8dd5-4bf6-84f1-8d04a4ae5cdf/volumes"
Oct 29 23:29:47.543072 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:47.543036    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"]
Oct 29 23:29:47.543580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:47.543273    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy" containerID="cri-o://121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45" gracePeriod=30
Oct 29 23:29:52.859400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:52.859371    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 23:29:53.007566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.007529    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-envoy\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.007753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.007594    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-podinfo\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.007753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.007640    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-token\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-token\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.007753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.007668    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-credential-socket\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.007753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.007696    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3128da-71cb-41a0-a907-f1c381ea16fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3128da-71cb-41a0-a907-f1c381ea16fa-istiod-ca-cert\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.007753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.007746    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-kube-api-access-tlcdk\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-kube-api-access-tlcdk\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.008077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.007937    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-certs\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.008077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008018    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-socket\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.008077 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008029    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-credential-socket" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:53.008251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008138    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3128da-71cb-41a0-a907-f1c381ea16fa-istiod-ca-cert" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:53.008251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008172    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-data\") pod \"fa3128da-71cb-41a0-a907-f1c381ea16fa\" (UID: \"fa3128da-71cb-41a0-a907-f1c381ea16fa\") "
Oct 29 23:29:53.008359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008271    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-certs" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:53.008468 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008445    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-socket" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:53.008535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008461    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-data" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:53.008748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008724    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.008923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008753    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.008923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008817    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.008923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008833    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.008923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.008847    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3128da-71cb-41a0-a907-f1c381ea16fa-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.010176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.010150    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-token" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:53.010304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.010245    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-podinfo" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:29:53.010304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.010263    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-envoy" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:29:53.010702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.010681    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-kube-api-access-tlcdk" pod "fa3128da-71cb-41a0-a907-f1c381ea16fa" (UID: "fa3128da-71cb-41a0-a907-f1c381ea16fa"). InnerVolumeSpecName "kube-api-access-tlcdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:53.039838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.039810    4827 generic.go:356] "Generic (PLEG): container finished" podID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerID="121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45" exitCode=0
Oct 29 23:29:53.039964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.039860    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"
Oct 29 23:29:53.039964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.039893    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" event={"ID":"fa3128da-71cb-41a0-a907-f1c381ea16fa","Type":"ContainerDied","Data":"121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45"}
Oct 29 23:29:53.039964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.039951    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db" event={"ID":"fa3128da-71cb-41a0-a907-f1c381ea16fa","Type":"ContainerDied","Data":"22f8c0ea579caa5f9ceac762fc35d991e91916cd425ff730cf0bfcb617c81f96"}
Oct 29 23:29:53.040081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.039983    4827 scope.go:122] "RemoveContainer" containerID="121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45"
Oct 29 23:29:53.050023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.050002    4827 scope.go:122] "RemoveContainer" containerID="4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d"
Oct 29 23:29:53.068313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.068292    4827 scope.go:122] "RemoveContainer" containerID="121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45"
Oct 29 23:29:53.068588 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:53.068572    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45\": container with ID starting with 121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45 not found: ID does not exist" containerID="121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45"
Oct 29 23:29:53.068636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.068595    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45"} err="failed to get container status \"121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45\": rpc error: code = NotFound desc = could not find container \"121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45\": container with ID starting with 121cf13dbfa78304ea45a05797371fc7334c7bf869769bbf4fdfa99f6fd33b45 not found: ID does not exist"
Oct 29 23:29:53.068636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.068612    4827 scope.go:122] "RemoveContainer" containerID="4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d"
Oct 29 23:29:53.068858 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:29:53.068841    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d\": container with ID starting with 4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d not found: ID does not exist" containerID="4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d"
Oct 29 23:29:53.068923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.068861    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d"} err="failed to get container status \"4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d\": rpc error: code = NotFound desc = could not find container \"4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d\": container with ID starting with 4093f88e4b372806c0c2445c621cd8e4971ed7b9c79873b4dc826e2c58a80d7d not found: ID does not exist"
Oct 29 23:29:53.089068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.089015    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"]
Oct 29 23:29:53.092795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.092755    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-6n5db"]
Oct 29 23:29:53.110399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.110375    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.110399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.110396    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.110531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.110405    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.110531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:53.110414    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlcdk\" (UniqueName: \"kubernetes.io/projected/fa3128da-71cb-41a0-a907-f1c381ea16fa-kube-api-access-tlcdk\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:29:54.492855 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:29:54.492819    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" path="/var/lib/kubelet/pods/fa3128da-71cb-41a0-a907-f1c381ea16fa/volumes"
Oct 29 23:30:00.143396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143304    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"]
Oct 29 23:30:00.143860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143846    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4203bcb2-2d10-4d67-ae1d-c7764f116179" containerName="revoke-keys"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143861    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4203bcb2-2d10-4d67-ae1d-c7764f116179" containerName="revoke-keys"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143873    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143878    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143892    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143896    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143905    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a9db7699-245e-4237-9301-096107be2fa2" containerName="payload-pre-processing"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143910    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9db7699-245e-4237-9301-096107be2fa2" containerName="payload-pre-processing"
Oct 29 23:30:00.143919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143921    4827 cpu_manager.go:414] "containerMap: removing container" podUID="09b13b1d-e33d-4399-bc4d-175bdafcee1c" containerName="revoke-keys"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143926    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b13b1d-e33d-4399-bc4d-175bdafcee1c" containerName="revoke-keys"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143933    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e87138d4-fac8-4de7-b5d0-021f38532334" containerName="maas-api"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143937    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87138d4-fac8-4de7-b5d0-021f38532334" containerName="maas-api"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143955    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.143960    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144043    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9db7699-245e-4237-9301-096107be2fa2" containerName="payload-pre-processing"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144052    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b13b1d-e33d-4399-bc4d-175bdafcee1c" containerName="revoke-keys"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144059    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87138d4-fac8-4de7-b5d0-021f38532334" containerName="maas-api"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144069    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144075    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144082    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4203bcb2-2d10-4d67-ae1d-c7764f116179" containerName="revoke-keys"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144088    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6022bfd-dfbd-45c3-8209-5697f8517b36" containerName="istio-proxy"
Oct 29 23:30:00.144228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.144095    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy"
Oct 29 23:30:00.149200 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.149170    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"]
Oct 29 23:30:00.149343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.149287    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.149958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.149932    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy"
Oct 29 23:30:00.149958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.149956    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3128da-71cb-41a0-a907-f1c381ea16fa" containerName="istio-proxy"
Oct 29 23:30:00.153344 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.153328    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.167214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.167193    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:30:00.170937 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.170913    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"]
Oct 29 23:30:00.176217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.176196    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.182196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.182172    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"]
Oct 29 23:30:00.187602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.187565    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"]
Oct 29 23:30:00.187730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.187702    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.192397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.192371    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"]
Oct 29 23:30:00.194987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.194965    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"]
Oct 29 23:30:00.200013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.199878    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.208827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.208798    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"]
Oct 29 23:30:00.212657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.212633    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"]
Oct 29 23:30:00.221719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.221693    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"]
Oct 29 23:30:00.306231 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306198    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.306419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306248    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njstd\" (UniqueName: \"kubernetes.io/projected/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-kube-api-access-njstd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.306419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306350    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcr2\" (UniqueName: \"kubernetes.io/projected/ebb1c9e2-a882-4337-86d6-6cf164554657-kube-api-access-8kcr2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.306501 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306456    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4500979a-4db1-44ce-8027-882a49580885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.306501 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306493    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.306590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306566    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1c9e2-a882-4337-86d6-6cf164554657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.306648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306624    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5f06e7-73db-48ae-987a-a62398a4b2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-6h6k9\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.306692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306657    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjtd\" (UniqueName: \"kubernetes.io/projected/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-kube-api-access-hwjtd\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.306740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306710    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6ll\" (UniqueName: \"kubernetes.io/projected/7d5f06e7-73db-48ae-987a-a62398a4b2b7-kube-api-access-db6ll\") pod \"maas-api-key-cleanup-29888610-6h6k9\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.306796 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.306751    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkq6h\" (UniqueName: \"kubernetes.io/projected/4500979a-4db1-44ce-8027-882a49580885-kube-api-access-mkq6h\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.407715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.407628    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkq6h\" (UniqueName: \"kubernetes.io/projected/4500979a-4db1-44ce-8027-882a49580885-kube-api-access-mkq6h\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.407715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.407684    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.407981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.407726    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njstd\" (UniqueName: \"kubernetes.io/projected/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-kube-api-access-njstd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.407981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.407801    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcr2\" (UniqueName: \"kubernetes.io/projected/ebb1c9e2-a882-4337-86d6-6cf164554657-kube-api-access-8kcr2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.407981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.407895    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4500979a-4db1-44ce-8027-882a49580885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.407981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.407950    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.408221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.407985    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1c9e2-a882-4337-86d6-6cf164554657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.408221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.408052    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5f06e7-73db-48ae-987a-a62398a4b2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-6h6k9\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.408221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.408111    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjtd\" (UniqueName: \"kubernetes.io/projected/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-kube-api-access-hwjtd\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.408221 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.408168    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6ll\" (UniqueName: \"kubernetes.io/projected/7d5f06e7-73db-48ae-987a-a62398a4b2b7-kube-api-access-db6ll\") pod \"maas-api-key-cleanup-29888610-6h6k9\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.408613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.408591    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.408757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.408732    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.408866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.408844    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1c9e2-a882-4337-86d6-6cf164554657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.409006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.408984    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4500979a-4db1-44ce-8027-882a49580885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.409130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.409030    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5f06e7-73db-48ae-987a-a62398a4b2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-6h6k9\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.430876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.430836    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjtd\" (UniqueName: \"kubernetes.io/projected/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-kube-api-access-hwjtd\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.431380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.431345    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcr2\" (UniqueName: \"kubernetes.io/projected/ebb1c9e2-a882-4337-86d6-6cf164554657-kube-api-access-8kcr2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.431538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.431383    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njstd\" (UniqueName: \"kubernetes.io/projected/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-kube-api-access-njstd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.431538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.431395    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkq6h\" (UniqueName: \"kubernetes.io/projected/4500979a-4db1-44ce-8027-882a49580885-kube-api-access-mkq6h\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.432359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.432337    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6ll\" (UniqueName: \"kubernetes.io/projected/7d5f06e7-73db-48ae-987a-a62398a4b2b7-kube-api-access-db6ll\") pod \"maas-api-key-cleanup-29888610-6h6k9\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.465369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.465343    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:00.473267 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.473245    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:00.493649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.493621    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:00.514005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.513976    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:00.532236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.531883    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:00.784682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.784655    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"]
Oct 29 23:30:00.786360 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:30:00.786325    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4500979a_4db1_44ce_8027_882a49580885.slice/crio-e7b8a4fd1e966c92f3082662dc078845914ba48f30d70bd8a40b442d73c05761 WatchSource:0}: Error finding container e7b8a4fd1e966c92f3082662dc078845914ba48f30d70bd8a40b442d73c05761: Status 404 returned error can't find the container with id e7b8a4fd1e966c92f3082662dc078845914ba48f30d70bd8a40b442d73c05761
Oct 29 23:30:00.798648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.798621    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"]
Oct 29 23:30:00.799079 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:30:00.799059    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4236ff5_dcd8_4ea8_b207_10491fa9ae6c.slice/crio-5990f8cef85a37468214933876b4e12f3effd48043feb792357d445c29397e82 WatchSource:0}: Error finding container 5990f8cef85a37468214933876b4e12f3effd48043feb792357d445c29397e82: Status 404 returned error can't find the container with id 5990f8cef85a37468214933876b4e12f3effd48043feb792357d445c29397e82
Oct 29 23:30:00.838271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.838247    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"]
Oct 29 23:30:00.842451 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:30:00.842161    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5f06e7_73db_48ae_987a_a62398a4b2b7.slice/crio-3466f42c34c065d29db07f7f37d20fe22b8e3059afc7aaa62b93cb2c07870776 WatchSource:0}: Error finding container 3466f42c34c065d29db07f7f37d20fe22b8e3059afc7aaa62b93cb2c07870776: Status 404 returned error can't find the container with id 3466f42c34c065d29db07f7f37d20fe22b8e3059afc7aaa62b93cb2c07870776
Oct 29 23:30:00.899227 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.898210    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"]
Oct 29 23:30:00.901235 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:00.901210    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"]
Oct 29 23:30:00.906314 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:30:00.906284    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dc02e0_b2a2_415b_8bfe_aefb65b3b68c.slice/crio-ec7cd77dc5d112438f8423494ed26189546aa4514674d247c707c4f013fdb584 WatchSource:0}: Error finding container ec7cd77dc5d112438f8423494ed26189546aa4514674d247c707c4f013fdb584: Status 404 returned error can't find the container with id ec7cd77dc5d112438f8423494ed26189546aa4514674d247c707c4f013fdb584
Oct 29 23:30:00.907160 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:30:00.907138    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb1c9e2_a882_4337_86d6_6cf164554657.slice/crio-f17bac8f4bc546921d8db7ce4dffbc46e4e0a8b75c9a7129420fcc37be4f9d92 WatchSource:0}: Error finding container f17bac8f4bc546921d8db7ce4dffbc46e4e0a8b75c9a7129420fcc37be4f9d92: Status 404 returned error can't find the container with id f17bac8f4bc546921d8db7ce4dffbc46e4e0a8b75c9a7129420fcc37be4f9d92
Oct 29 23:30:01.092554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.092529    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" event={"ID":"ebb1c9e2-a882-4337-86d6-6cf164554657","Type":"ContainerStarted","Data":"f17bac8f4bc546921d8db7ce4dffbc46e4e0a8b75c9a7129420fcc37be4f9d92"}
Oct 29 23:30:01.094126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.094103    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" event={"ID":"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c","Type":"ContainerStarted","Data":"ceaf70e9cf4e844f9e01f16183dbf983a9847567e35461e986d274cfc2126311"}
Oct 29 23:30:01.094251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.094151    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" event={"ID":"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c","Type":"ContainerStarted","Data":"5990f8cef85a37468214933876b4e12f3effd48043feb792357d445c29397e82"}
Oct 29 23:30:01.095449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.095424    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" event={"ID":"7d5f06e7-73db-48ae-987a-a62398a4b2b7","Type":"ContainerStarted","Data":"3466f42c34c065d29db07f7f37d20fe22b8e3059afc7aaa62b93cb2c07870776"}
Oct 29 23:30:01.096874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.096853    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" event={"ID":"4500979a-4db1-44ce-8027-882a49580885","Type":"ContainerStarted","Data":"5baf3abd74ef7914a58eec02921929207adbed88f0f2708bfe1bd4ae9b1fd9bb"}
Oct 29 23:30:01.096990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.096893    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" event={"ID":"4500979a-4db1-44ce-8027-882a49580885","Type":"ContainerStarted","Data":"e7b8a4fd1e966c92f3082662dc078845914ba48f30d70bd8a40b442d73c05761"}
Oct 29 23:30:01.097995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.097967    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" event={"ID":"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c","Type":"ContainerStarted","Data":"ec7cd77dc5d112438f8423494ed26189546aa4514674d247c707c4f013fdb584"}
Oct 29 23:30:01.131757 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.131720    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" podStartSLOduration=1.131708353 podStartE2EDuration="1.131708353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.130999215 +0000 UTC m=+2836.753910235" watchObservedRunningTime="2026-10-29 23:30:01.131708353 +0000 UTC m=+2836.754619373"
Oct 29 23:30:01.168647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:01.168602    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" podStartSLOduration=1.168587545 podStartE2EDuration="1.168587545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.16788831 +0000 UTC m=+2836.790799358" watchObservedRunningTime="2026-10-29 23:30:01.168587545 +0000 UTC m=+2836.791498565"
Oct 29 23:30:02.158799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.151838    4827 generic.go:356] "Generic (PLEG): container finished" podID="4500979a-4db1-44ce-8027-882a49580885" containerID="5baf3abd74ef7914a58eec02921929207adbed88f0f2708bfe1bd4ae9b1fd9bb" exitCode=60
Oct 29 23:30:02.158799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.151992    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" event={"ID":"4500979a-4db1-44ce-8027-882a49580885","Type":"ContainerDied","Data":"5baf3abd74ef7914a58eec02921929207adbed88f0f2708bfe1bd4ae9b1fd9bb"}
Oct 29 23:30:02.158799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.152507    4827 scope.go:122] "RemoveContainer" containerID="5baf3abd74ef7914a58eec02921929207adbed88f0f2708bfe1bd4ae9b1fd9bb"
Oct 29 23:30:02.177657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.176219    4827 generic.go:356] "Generic (PLEG): container finished" podID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerID="6d402f1f42add19a5300625b164f8f084fb03089b3238306bfe51e94ff665a5f" exitCode=60
Oct 29 23:30:02.177657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.176393    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" event={"ID":"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c","Type":"ContainerDied","Data":"6d402f1f42add19a5300625b164f8f084fb03089b3238306bfe51e94ff665a5f"}
Oct 29 23:30:02.177657 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.176905    4827 scope.go:122] "RemoveContainer" containerID="6d402f1f42add19a5300625b164f8f084fb03089b3238306bfe51e94ff665a5f"
Oct 29 23:30:02.187636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.185911    4827 generic.go:356] "Generic (PLEG): container finished" podID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerID="69059c59b932bd1a56c0ecf35d44ddb11a95b95810d98765308b10e2db2763f8" exitCode=60
Oct 29 23:30:02.187636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.186032    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" event={"ID":"ebb1c9e2-a882-4337-86d6-6cf164554657","Type":"ContainerDied","Data":"69059c59b932bd1a56c0ecf35d44ddb11a95b95810d98765308b10e2db2763f8"}
Oct 29 23:30:02.187636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.186446    4827 scope.go:122] "RemoveContainer" containerID="69059c59b932bd1a56c0ecf35d44ddb11a95b95810d98765308b10e2db2763f8"
Oct 29 23:30:02.191626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.190267    4827 generic.go:356] "Generic (PLEG): container finished" podID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerID="ceaf70e9cf4e844f9e01f16183dbf983a9847567e35461e986d274cfc2126311" exitCode=60
Oct 29 23:30:02.191626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.190394    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" event={"ID":"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c","Type":"ContainerDied","Data":"ceaf70e9cf4e844f9e01f16183dbf983a9847567e35461e986d274cfc2126311"}
Oct 29 23:30:02.191626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.190801    4827 scope.go:122] "RemoveContainer" containerID="ceaf70e9cf4e844f9e01f16183dbf983a9847567e35461e986d274cfc2126311"
Oct 29 23:30:02.199562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.199094    4827 generic.go:356] "Generic (PLEG): container finished" podID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerID="bbda2f64c731d01121985f116e1be899590505f435e704f9bd017f2815f75be1" exitCode=60
Oct 29 23:30:02.199562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.199147    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" event={"ID":"7d5f06e7-73db-48ae-987a-a62398a4b2b7","Type":"ContainerDied","Data":"bbda2f64c731d01121985f116e1be899590505f435e704f9bd017f2815f75be1"}
Oct 29 23:30:02.199562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:02.199418    4827 scope.go:122] "RemoveContainer" containerID="bbda2f64c731d01121985f116e1be899590505f435e704f9bd017f2815f75be1"
Oct 29 23:30:02.827277 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:02.827229    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:30:03.207645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.207614    4827 generic.go:356] "Generic (PLEG): container finished" podID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerID="00176dc99d89ed9dbe5e3515511349572b4ab2d120604e6d619484d4ab79cad3" exitCode=60
Oct 29 23:30:03.208148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.207707    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" event={"ID":"7d5f06e7-73db-48ae-987a-a62398a4b2b7","Type":"ContainerDied","Data":"00176dc99d89ed9dbe5e3515511349572b4ab2d120604e6d619484d4ab79cad3"}
Oct 29 23:30:03.208148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.207879    4827 scope.go:122] "RemoveContainer" containerID="bbda2f64c731d01121985f116e1be899590505f435e704f9bd017f2815f75be1"
Oct 29 23:30:03.208148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.208075    4827 scope.go:122] "RemoveContainer" containerID="00176dc99d89ed9dbe5e3515511349572b4ab2d120604e6d619484d4ab79cad3"
Oct 29 23:30:03.208473 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:03.208444    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-6h6k9_odh-ai-gateway-infra(7d5f06e7-73db-48ae-987a-a62398a4b2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7"
Oct 29 23:30:03.209984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.209962    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" event={"ID":"4500979a-4db1-44ce-8027-882a49580885","Type":"ContainerStarted","Data":"fd3c0545a0f9a214414e0d6169f4cb5534344d492a73ad559a538aaa8792c4c1"}
Oct 29 23:30:03.211842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.211820    4827 generic.go:356] "Generic (PLEG): container finished" podID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerID="edb00387a4def9fa496a1ba85ba5a51c7335348cd48ceff3b9c5adc98df974dc" exitCode=60
Oct 29 23:30:03.212002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.211901    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" event={"ID":"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c","Type":"ContainerDied","Data":"edb00387a4def9fa496a1ba85ba5a51c7335348cd48ceff3b9c5adc98df974dc"}
Oct 29 23:30:03.212177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.212158    4827 scope.go:122] "RemoveContainer" containerID="edb00387a4def9fa496a1ba85ba5a51c7335348cd48ceff3b9c5adc98df974dc"
Oct 29 23:30:03.212472 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:03.212454    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x_odh-ai-gateway-infra(54dc02e0-b2a2-415b-8bfe-aefb65b3b68c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c"
Oct 29 23:30:03.214248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.214230    4827 generic.go:356] "Generic (PLEG): container finished" podID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerID="cad002f3b87191f297fcf262bac5dfab44e19e4fcd2901c6f51019ab6bd8ff26" exitCode=60
Oct 29 23:30:03.214358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.214323    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" event={"ID":"ebb1c9e2-a882-4337-86d6-6cf164554657","Type":"ContainerDied","Data":"cad002f3b87191f297fcf262bac5dfab44e19e4fcd2901c6f51019ab6bd8ff26"}
Oct 29 23:30:03.214590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.214573    4827 scope.go:122] "RemoveContainer" containerID="cad002f3b87191f297fcf262bac5dfab44e19e4fcd2901c6f51019ab6bd8ff26"
Oct 29 23:30:03.214885 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:03.214867    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm_odh-ai-gateway-infra(ebb1c9e2-a882-4337-86d6-6cf164554657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657"
Oct 29 23:30:03.216191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.216170    4827 generic.go:356] "Generic (PLEG): container finished" podID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerID="4b3706f36ab44aa878d66a17b19a4e047703fd36e33d36830af5e767ecb9f2aa" exitCode=60
Oct 29 23:30:03.216284 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.216209    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" event={"ID":"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c","Type":"ContainerDied","Data":"4b3706f36ab44aa878d66a17b19a4e047703fd36e33d36830af5e767ecb9f2aa"}
Oct 29 23:30:03.216408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.216395    4827 scope.go:122] "RemoveContainer" containerID="4b3706f36ab44aa878d66a17b19a4e047703fd36e33d36830af5e767ecb9f2aa"
Oct 29 23:30:03.216634 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:03.216619    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq_odh-ai-gateway-infra(a4236ff5-dcd8-4ea8-b207-10491fa9ae6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c"
Oct 29 23:30:03.341610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.341585    4827 scope.go:122] "RemoveContainer" containerID="6d402f1f42add19a5300625b164f8f084fb03089b3238306bfe51e94ff665a5f"
Oct 29 23:30:03.354990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.354967    4827 scope.go:122] "RemoveContainer" containerID="69059c59b932bd1a56c0ecf35d44ddb11a95b95810d98765308b10e2db2763f8"
Oct 29 23:30:03.368409 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:03.368385    4827 scope.go:122] "RemoveContainer" containerID="ceaf70e9cf4e844f9e01f16183dbf983a9847567e35461e986d274cfc2126311"
Oct 29 23:30:04.225340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.225312    4827 scope.go:122] "RemoveContainer" containerID="cad002f3b87191f297fcf262bac5dfab44e19e4fcd2901c6f51019ab6bd8ff26"
Oct 29 23:30:04.225815 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:04.225632    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm_odh-ai-gateway-infra(ebb1c9e2-a882-4337-86d6-6cf164554657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657"
Oct 29 23:30:04.226985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.226967    4827 scope.go:122] "RemoveContainer" containerID="4b3706f36ab44aa878d66a17b19a4e047703fd36e33d36830af5e767ecb9f2aa"
Oct 29 23:30:04.227292 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:04.227268    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq_odh-ai-gateway-infra(a4236ff5-dcd8-4ea8-b207-10491fa9ae6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c"
Oct 29 23:30:04.228554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.228534    4827 scope.go:122] "RemoveContainer" containerID="00176dc99d89ed9dbe5e3515511349572b4ab2d120604e6d619484d4ab79cad3"
Oct 29 23:30:04.228826 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:04.228808    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-6h6k9_odh-ai-gateway-infra(7d5f06e7-73db-48ae-987a-a62398a4b2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7"
Oct 29 23:30:04.230104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.230082    4827 generic.go:356] "Generic (PLEG): container finished" podID="4500979a-4db1-44ce-8027-882a49580885" containerID="fd3c0545a0f9a214414e0d6169f4cb5534344d492a73ad559a538aaa8792c4c1" exitCode=60
Oct 29 23:30:04.230188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.230134    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" event={"ID":"4500979a-4db1-44ce-8027-882a49580885","Type":"ContainerDied","Data":"fd3c0545a0f9a214414e0d6169f4cb5534344d492a73ad559a538aaa8792c4c1"}
Oct 29 23:30:04.230188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.230183    4827 scope.go:122] "RemoveContainer" containerID="5baf3abd74ef7914a58eec02921929207adbed88f0f2708bfe1bd4ae9b1fd9bb"
Oct 29 23:30:04.230390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.230372    4827 scope.go:122] "RemoveContainer" containerID="fd3c0545a0f9a214414e0d6169f4cb5534344d492a73ad559a538aaa8792c4c1"
Oct 29 23:30:04.230631 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:04.230607    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw_odh-ai-gateway-infra(4500979a-4db1-44ce-8027-882a49580885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" podUID="4500979a-4db1-44ce-8027-882a49580885"
Oct 29 23:30:04.232194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:04.232177    4827 scope.go:122] "RemoveContainer" containerID="edb00387a4def9fa496a1ba85ba5a51c7335348cd48ceff3b9c5adc98df974dc"
Oct 29 23:30:04.232418 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:04.232396    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x_odh-ai-gateway-infra(54dc02e0-b2a2-415b-8bfe-aefb65b3b68c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c"
Oct 29 23:30:05.240282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:05.240242    4827 scope.go:122] "RemoveContainer" containerID="fd3c0545a0f9a214414e0d6169f4cb5534344d492a73ad559a538aaa8792c4c1"
Oct 29 23:30:05.240790 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:05.240553    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw_odh-ai-gateway-infra(4500979a-4db1-44ce-8027-882a49580885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" podUID="4500979a-4db1-44ce-8027-882a49580885"
Oct 29 23:30:14.484108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:14.484078    4827 scope.go:122] "RemoveContainer" containerID="edb00387a4def9fa496a1ba85ba5a51c7335348cd48ceff3b9c5adc98df974dc"
Oct 29 23:30:14.484581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:14.484189    4827 scope.go:122] "RemoveContainer" containerID="cad002f3b87191f297fcf262bac5dfab44e19e4fcd2901c6f51019ab6bd8ff26"
Oct 29 23:30:15.310270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:15.310240    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" event={"ID":"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c","Type":"ContainerStarted","Data":"8de7405cc26efe74f96932407233a0a313823a45020878fb7faa469e2e642352"}
Oct 29 23:30:15.312174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:15.312141    4827 generic.go:356] "Generic (PLEG): container finished" podID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerID="d12944247dd9a8e915059f30f8e21e7786784f78a73c6e47b2bf2c34153d90e1" exitCode=60
Oct 29 23:30:15.312318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:15.312186    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" event={"ID":"ebb1c9e2-a882-4337-86d6-6cf164554657","Type":"ContainerDied","Data":"d12944247dd9a8e915059f30f8e21e7786784f78a73c6e47b2bf2c34153d90e1"}
Oct 29 23:30:15.312318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:15.312260    4827 scope.go:122] "RemoveContainer" containerID="cad002f3b87191f297fcf262bac5dfab44e19e4fcd2901c6f51019ab6bd8ff26"
Oct 29 23:30:15.312624 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:15.312608    4827 scope.go:122] "RemoveContainer" containerID="d12944247dd9a8e915059f30f8e21e7786784f78a73c6e47b2bf2c34153d90e1"
Oct 29 23:30:15.312952 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:15.312933    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm_odh-ai-gateway-infra(ebb1c9e2-a882-4337-86d6-6cf164554657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657"
Oct 29 23:30:15.349649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:15.349602    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" podStartSLOduration=15.349587803 podStartE2EDuration="15.349587803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:15.346957079 +0000 UTC m=+2850.969868100" watchObservedRunningTime="2026-10-29 23:30:15.349587803 +0000 UTC m=+2850.972498823"
Oct 29 23:30:15.480965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:15.480937    4827 scope.go:122] "RemoveContainer" containerID="00176dc99d89ed9dbe5e3515511349572b4ab2d120604e6d619484d4ab79cad3"
Oct 29 23:30:16.322833 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.322798    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" event={"ID":"7d5f06e7-73db-48ae-987a-a62398a4b2b7","Type":"ContainerStarted","Data":"41b3944aa9d656439044aa1af266c737e6d118162ddc1d1f3fc6372214c1f114"}
Oct 29 23:30:16.324698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.324672    4827 generic.go:356] "Generic (PLEG): container finished" podID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerID="8de7405cc26efe74f96932407233a0a313823a45020878fb7faa469e2e642352" exitCode=60
Oct 29 23:30:16.324864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.324740    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" event={"ID":"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c","Type":"ContainerDied","Data":"8de7405cc26efe74f96932407233a0a313823a45020878fb7faa469e2e642352"}
Oct 29 23:30:16.324864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.324826    4827 scope.go:122] "RemoveContainer" containerID="edb00387a4def9fa496a1ba85ba5a51c7335348cd48ceff3b9c5adc98df974dc"
Oct 29 23:30:16.325132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.325118    4827 scope.go:122] "RemoveContainer" containerID="8de7405cc26efe74f96932407233a0a313823a45020878fb7faa469e2e642352"
Oct 29 23:30:16.325407 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:16.325389    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x_odh-ai-gateway-infra(54dc02e0-b2a2-415b-8bfe-aefb65b3b68c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c"
Oct 29 23:30:16.358138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.358111    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"]
Oct 29 23:30:16.361289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.361249    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" podStartSLOduration=16.361238065 podStartE2EDuration="16.361238065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:16.360121 +0000 UTC m=+2851.983032025" watchObservedRunningTime="2026-10-29 23:30:16.361238065 +0000 UTC m=+2851.984149085"
Oct 29 23:30:16.394053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.394028    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"]
Oct 29 23:30:16.481605 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.481581    4827 scope.go:122] "RemoveContainer" containerID="fd3c0545a0f9a214414e0d6169f4cb5534344d492a73ad559a538aaa8792c4c1"
Oct 29 23:30:16.548815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.548744    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:16.603229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.603200    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb1c9e2-a882-4337-86d6-6cf164554657-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1c9e2-a882-4337-86d6-6cf164554657-maas-api-service-ca\") pod \"ebb1c9e2-a882-4337-86d6-6cf164554657\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") "
Oct 29 23:30:16.603422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.603356    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb1c9e2-a882-4337-86d6-6cf164554657-kube-api-access-8kcr2\" (UniqueName: \"kubernetes.io/projected/ebb1c9e2-a882-4337-86d6-6cf164554657-kube-api-access-8kcr2\") pod \"ebb1c9e2-a882-4337-86d6-6cf164554657\" (UID: \"ebb1c9e2-a882-4337-86d6-6cf164554657\") "
Oct 29 23:30:16.603613 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.603576    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb1c9e2-a882-4337-86d6-6cf164554657-maas-api-service-ca" pod "ebb1c9e2-a882-4337-86d6-6cf164554657" (UID: "ebb1c9e2-a882-4337-86d6-6cf164554657"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:16.603836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.603818    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1c9e2-a882-4337-86d6-6cf164554657-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:16.605475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.605450    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb1c9e2-a882-4337-86d6-6cf164554657-kube-api-access-8kcr2" pod "ebb1c9e2-a882-4337-86d6-6cf164554657" (UID: "ebb1c9e2-a882-4337-86d6-6cf164554657"). InnerVolumeSpecName "kube-api-access-8kcr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:16.705322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:16.705297    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcr2\" (UniqueName: \"kubernetes.io/projected/ebb1c9e2-a882-4337-86d6-6cf164554657-kube-api-access-8kcr2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:17.332695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.332665    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"
Oct 29 23:30:17.333170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.332696    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm" event={"ID":"ebb1c9e2-a882-4337-86d6-6cf164554657","Type":"ContainerDied","Data":"f17bac8f4bc546921d8db7ce4dffbc46e4e0a8b75c9a7129420fcc37be4f9d92"}
Oct 29 23:30:17.333170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.332755    4827 scope.go:122] "RemoveContainer" containerID="d12944247dd9a8e915059f30f8e21e7786784f78a73c6e47b2bf2c34153d90e1"
Oct 29 23:30:17.335599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.335571    4827 generic.go:356] "Generic (PLEG): container finished" podID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerID="41b3944aa9d656439044aa1af266c737e6d118162ddc1d1f3fc6372214c1f114" exitCode=60
Oct 29 23:30:17.335755 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.335615    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" event={"ID":"7d5f06e7-73db-48ae-987a-a62398a4b2b7","Type":"ContainerDied","Data":"41b3944aa9d656439044aa1af266c737e6d118162ddc1d1f3fc6372214c1f114"}
Oct 29 23:30:17.336040 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.336021    4827 scope.go:122] "RemoveContainer" containerID="41b3944aa9d656439044aa1af266c737e6d118162ddc1d1f3fc6372214c1f114"
Oct 29 23:30:17.336339 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:17.336320    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-6h6k9_odh-ai-gateway-infra(7d5f06e7-73db-48ae-987a-a62398a4b2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7"
Oct 29 23:30:17.337672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.337648    4827 generic.go:356] "Generic (PLEG): container finished" podID="4500979a-4db1-44ce-8027-882a49580885" containerID="9420911bfd411173a0236b0c43bc53bbcab364304cd301e9686782c49f0dd256" exitCode=60
Oct 29 23:30:17.337807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.337715    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" event={"ID":"4500979a-4db1-44ce-8027-882a49580885","Type":"ContainerDied","Data":"9420911bfd411173a0236b0c43bc53bbcab364304cd301e9686782c49f0dd256"}
Oct 29 23:30:17.338139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.338121    4827 scope.go:122] "RemoveContainer" containerID="9420911bfd411173a0236b0c43bc53bbcab364304cd301e9686782c49f0dd256"
Oct 29 23:30:17.338453 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:17.338433    4827 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw_odh-ai-gateway-infra(4500979a-4db1-44ce-8027-882a49580885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" podUID="4500979a-4db1-44ce-8027-882a49580885"
Oct 29 23:30:17.344632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.344618    4827 scope.go:122] "RemoveContainer" containerID="00176dc99d89ed9dbe5e3515511349572b4ab2d120604e6d619484d4ab79cad3"
Oct 29 23:30:17.359337 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.359308    4827 scope.go:122] "RemoveContainer" containerID="fd3c0545a0f9a214414e0d6169f4cb5534344d492a73ad559a538aaa8792c4c1"
Oct 29 23:30:17.374572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.374549    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"]
Oct 29 23:30:17.467058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.467025    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"]
Oct 29 23:30:17.479580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.479553    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888610-hkbcm"]
Oct 29 23:30:17.481058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.481038    4827 scope.go:122] "RemoveContainer" containerID="4b3706f36ab44aa878d66a17b19a4e047703fd36e33d36830af5e767ecb9f2aa"
Oct 29 23:30:17.542184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.542162    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:17.615674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.615635    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-maas-api-service-ca\") pod \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") "
Oct 29 23:30:17.615905 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.615743    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-kube-api-access-hwjtd\" (UniqueName: \"kubernetes.io/projected/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-kube-api-access-hwjtd\") pod \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\" (UID: \"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c\") "
Oct 29 23:30:17.616076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.616053    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-maas-api-service-ca" pod "54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" (UID: "54dc02e0-b2a2-415b-8bfe-aefb65b3b68c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:17.616453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.616433    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:17.617981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.617945    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-kube-api-access-hwjtd" pod "54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" (UID: "54dc02e0-b2a2-415b-8bfe-aefb65b3b68c"). InnerVolumeSpecName "kube-api-access-hwjtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:17.718216 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:17.718187    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjtd\" (UniqueName: \"kubernetes.io/projected/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c-kube-api-access-hwjtd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:18.349188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.349150    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" event={"ID":"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c","Type":"ContainerStarted","Data":"d43fd958b6970604b12409086c14ace51ff5a63762f948a042140cad6570664c"}
Oct 29 23:30:18.353807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.353779    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x" event={"ID":"54dc02e0-b2a2-415b-8bfe-aefb65b3b68c","Type":"ContainerDied","Data":"ec7cd77dc5d112438f8423494ed26189546aa4514674d247c707c4f013fdb584"}
Oct 29 23:30:18.353807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.353799    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"
Oct 29 23:30:18.354000 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.353836    4827 scope.go:122] "RemoveContainer" containerID="8de7405cc26efe74f96932407233a0a313823a45020878fb7faa469e2e642352"
Oct 29 23:30:18.400349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.400317    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"]
Oct 29 23:30:18.502424 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.502261    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" path="/var/lib/kubelet/pods/ebb1c9e2-a882-4337-86d6-6cf164554657/volumes"
Oct 29 23:30:18.526816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.525967    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:18.534806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.533879    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"]
Oct 29 23:30:18.542003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.541979    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888610-d7p9x"]
Oct 29 23:30:18.596584 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.596557    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:18.653792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.652669    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4500979a-4db1-44ce-8027-882a49580885-kube-api-access-mkq6h\" (UniqueName: \"kubernetes.io/projected/4500979a-4db1-44ce-8027-882a49580885-kube-api-access-mkq6h\") pod \"4500979a-4db1-44ce-8027-882a49580885\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") "
Oct 29 23:30:18.653792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.652809    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5f06e7-73db-48ae-987a-a62398a4b2b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5f06e7-73db-48ae-987a-a62398a4b2b7-maas-api-service-ca\") pod \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") "
Oct 29 23:30:18.653792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.652889    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4500979a-4db1-44ce-8027-882a49580885-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4500979a-4db1-44ce-8027-882a49580885-maas-api-service-ca\") pod \"4500979a-4db1-44ce-8027-882a49580885\" (UID: \"4500979a-4db1-44ce-8027-882a49580885\") "
Oct 29 23:30:18.653792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.652971    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5f06e7-73db-48ae-987a-a62398a4b2b7-kube-api-access-db6ll\" (UniqueName: \"kubernetes.io/projected/7d5f06e7-73db-48ae-987a-a62398a4b2b7-kube-api-access-db6ll\") pod \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\" (UID: \"7d5f06e7-73db-48ae-987a-a62398a4b2b7\") "
Oct 29 23:30:18.653792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.653475    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5f06e7-73db-48ae-987a-a62398a4b2b7-maas-api-service-ca" pod "7d5f06e7-73db-48ae-987a-a62398a4b2b7" (UID: "7d5f06e7-73db-48ae-987a-a62398a4b2b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:18.654213 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.653991    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4500979a-4db1-44ce-8027-882a49580885-maas-api-service-ca" pod "4500979a-4db1-44ce-8027-882a49580885" (UID: "4500979a-4db1-44ce-8027-882a49580885"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:18.655999 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.655378    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4500979a-4db1-44ce-8027-882a49580885-kube-api-access-mkq6h" pod "4500979a-4db1-44ce-8027-882a49580885" (UID: "4500979a-4db1-44ce-8027-882a49580885"). InnerVolumeSpecName "kube-api-access-mkq6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:18.659270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.659241    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5f06e7-73db-48ae-987a-a62398a4b2b7-kube-api-access-db6ll" pod "7d5f06e7-73db-48ae-987a-a62398a4b2b7" (UID: "7d5f06e7-73db-48ae-987a-a62398a4b2b7"). InnerVolumeSpecName "kube-api-access-db6ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:18.754982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.754950    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5f06e7-73db-48ae-987a-a62398a4b2b7-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:18.754982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.754982    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4500979a-4db1-44ce-8027-882a49580885-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:18.755188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.754992    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db6ll\" (UniqueName: \"kubernetes.io/projected/7d5f06e7-73db-48ae-987a-a62398a4b2b7-kube-api-access-db6ll\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:18.755188 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:18.755001    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkq6h\" (UniqueName: \"kubernetes.io/projected/4500979a-4db1-44ce-8027-882a49580885-kube-api-access-mkq6h\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:19.393595 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.393511    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9" event={"ID":"7d5f06e7-73db-48ae-987a-a62398a4b2b7","Type":"ContainerDied","Data":"3466f42c34c065d29db07f7f37d20fe22b8e3059afc7aaa62b93cb2c07870776"}
Oct 29 23:30:19.402010 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.398196    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"
Oct 29 23:30:19.402810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.399011    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"
Oct 29 23:30:19.413360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.409303    4827 generic.go:356] "Generic (PLEG): container finished" podID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerID="d43fd958b6970604b12409086c14ace51ff5a63762f948a042140cad6570664c" exitCode=60
Oct 29 23:30:19.413741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.413707    4827 scope.go:122] "RemoveContainer" containerID="41b3944aa9d656439044aa1af266c737e6d118162ddc1d1f3fc6372214c1f114"
Oct 29 23:30:19.417350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.417315    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw" event={"ID":"4500979a-4db1-44ce-8027-882a49580885","Type":"ContainerDied","Data":"e7b8a4fd1e966c92f3082662dc078845914ba48f30d70bd8a40b442d73c05761"}
Oct 29 23:30:19.417485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.417421    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" event={"ID":"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c","Type":"ContainerDied","Data":"d43fd958b6970604b12409086c14ace51ff5a63762f948a042140cad6570664c"}
Oct 29 23:30:19.417485 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.417475    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"]
Oct 29 23:30:19.431461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.431445    4827 scope.go:122] "RemoveContainer" containerID="9420911bfd411173a0236b0c43bc53bbcab364304cd301e9686782c49f0dd256"
Oct 29 23:30:19.443945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.443927    4827 scope.go:122] "RemoveContainer" containerID="4b3706f36ab44aa878d66a17b19a4e047703fd36e33d36830af5e767ecb9f2aa"
Oct 29 23:30:19.470967 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.470945    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"]
Oct 29 23:30:19.485196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.483719    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-65e2d3-29888610-wqxrw"]
Oct 29 23:30:19.536889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.536855    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"]
Oct 29 23:30:19.539888 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.539865    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-6h6k9"]
Oct 29 23:30:19.593914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.593890    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:19.666566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.666489    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-kube-api-access-njstd\" (UniqueName: \"kubernetes.io/projected/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-kube-api-access-njstd\") pod \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") "
Oct 29 23:30:19.666735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.666584    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-maas-api-service-ca\") pod \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\" (UID: \"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c\") "
Oct 29 23:30:19.667037 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.667012    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-maas-api-service-ca" pod "a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" (UID: "a4236ff5-dcd8-4ea8-b207-10491fa9ae6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:19.667164 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.667148    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-maas-api-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:19.669281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.669251    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-kube-api-access-njstd" pod "a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" (UID: "a4236ff5-dcd8-4ea8-b207-10491fa9ae6c"). InnerVolumeSpecName "kube-api-access-njstd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:19.768407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:19.768373    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njstd\" (UniqueName: \"kubernetes.io/projected/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c-kube-api-access-njstd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:30:20.417645 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.417615    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"
Oct 29 23:30:20.418184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.417620    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq" event={"ID":"a4236ff5-dcd8-4ea8-b207-10491fa9ae6c","Type":"ContainerDied","Data":"5990f8cef85a37468214933876b4e12f3effd48043feb792357d445c29397e82"}
Oct 29 23:30:20.418184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.417801    4827 scope.go:122] "RemoveContainer" containerID="d43fd958b6970604b12409086c14ace51ff5a63762f948a042140cad6570664c"
Oct 29 23:30:20.525587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.521775    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4500979a-4db1-44ce-8027-882a49580885" path="/var/lib/kubelet/pods/4500979a-4db1-44ce-8027-882a49580885/volumes"
Oct 29 23:30:20.525587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.522517    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" path="/var/lib/kubelet/pods/54dc02e0-b2a2-415b-8bfe-aefb65b3b68c/volumes"
Oct 29 23:30:20.525587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.523146    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" path="/var/lib/kubelet/pods/7d5f06e7-73db-48ae-987a-a62398a4b2b7/volumes"
Oct 29 23:30:20.525587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.523989    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"]
Oct 29 23:30:20.526821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:20.526794    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888610-gmbbq"]
Oct 29 23:30:22.488968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:30:22.488936    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" path="/var/lib/kubelet/pods/a4236ff5-dcd8-4ea8-b207-10491fa9ae6c/volumes"
Oct 29 23:30:32.824340 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:30:32.824303    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:31:02.823408 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:31:02.823364    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:31:03.664853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:03.663014    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"]
Oct 29 23:31:03.664853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:03.663295    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy" containerID="cri-o://7b1af9d45ddba4568f35fb8bbe7f5d458639ca64fedd0b2ad8768a7137b23636" gracePeriod=30
Oct 29 23:31:04.233486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.233452    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"]
Oct 29 23:31:04.234217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234196    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234219    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234242    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234250    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234262    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234270    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234282    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234289    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234301    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234308    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:04.234324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234328    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234335    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234345    4827 cpu_manager.go:414] "containerMap: removing container" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234352    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234364    4827 cpu_manager.go:414] "containerMap: removing container" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234371    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234387    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234394    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234420    4827 cpu_manager.go:414] "containerMap: removing container" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234428    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234455    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234463    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234476    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234482    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234494    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234500    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234634    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234648    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234657    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234666    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234674    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234683    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234691    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4500979a-4db1-44ce-8027-882a49580885" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234702    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234710    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234718    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:04.234910 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.234729    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:04.240244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.240223    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.257092 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.257071    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"]
Oct 29 23:31:04.312802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.312737    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:31:04.312802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.312801    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:04.386747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.386708    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxmh\" (UniqueName: \"kubernetes.io/projected/3aa8284d-404b-4096-b89f-40a8cf493f51-kube-api-access-vrxmh\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.386936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.386858    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.386936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.386926    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.488674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.488585    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxmh\" (UniqueName: \"kubernetes.io/projected/3aa8284d-404b-4096-b89f-40a8cf493f51-kube-api-access-vrxmh\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.488674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.488663    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.488951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.488695    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.489320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.489294    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.491283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.491264    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.510343 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.510313    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxmh\" (UniqueName: \"kubernetes.io/projected/3aa8284d-404b-4096-b89f-40a8cf493f51-kube-api-access-vrxmh\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.553950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.553920    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:04.857928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.857900    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"]
Oct 29 23:31:04.859494 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:31:04.859467    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa8284d_404b_4096_b89f_40a8cf493f51.slice/crio-c970d4392a01d29e75cb2aea793b9cf64a08ad77fb4da6ec4093e4d86f265f3c WatchSource:0}: Error finding container c970d4392a01d29e75cb2aea793b9cf64a08ad77fb4da6ec4093e4d86f265f3c: Status 404 returned error can't find the container with id c970d4392a01d29e75cb2aea793b9cf64a08ad77fb4da6ec4093e4d86f265f3c
Oct 29 23:31:04.885621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:04.885592    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j" event={"ID":"3aa8284d-404b-4096-b89f-40a8cf493f51","Type":"ContainerStarted","Data":"c970d4392a01d29e75cb2aea793b9cf64a08ad77fb4da6ec4093e4d86f265f3c"}
Oct 29 23:31:05.893116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:05.893081    4827 generic.go:356] "Generic (PLEG): container finished" podID="3aa8284d-404b-4096-b89f-40a8cf493f51" containerID="70ae7e775509099fba314bf3f3bdf1fb03a05050a9905b4fbfc8d2923dfc0796" exitCode=0
Oct 29 23:31:05.893507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:05.893160    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j" event={"ID":"3aa8284d-404b-4096-b89f-40a8cf493f51","Type":"ContainerDied","Data":"70ae7e775509099fba314bf3f3bdf1fb03a05050a9905b4fbfc8d2923dfc0796"}
Oct 29 23:31:07.059506 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.059476    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:07.119414 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.119373    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-xks-ca\") pod \"3aa8284d-404b-4096-b89f-40a8cf493f51\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") "
Oct 29 23:31:07.119691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.119445    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-openshift-ca\") pod \"3aa8284d-404b-4096-b89f-40a8cf493f51\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") "
Oct 29 23:31:07.119691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.119478    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa8284d-404b-4096-b89f-40a8cf493f51-kube-api-access-vrxmh\" (UniqueName: \"kubernetes.io/projected/3aa8284d-404b-4096-b89f-40a8cf493f51-kube-api-access-vrxmh\") pod \"3aa8284d-404b-4096-b89f-40a8cf493f51\" (UID: \"3aa8284d-404b-4096-b89f-40a8cf493f51\") "
Oct 29 23:31:07.119870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.119836    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-openshift-ca" pod "3aa8284d-404b-4096-b89f-40a8cf493f51" (UID: "3aa8284d-404b-4096-b89f-40a8cf493f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:07.120123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.120101    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:07.121691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.121670    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa8284d-404b-4096-b89f-40a8cf493f51-kube-api-access-vrxmh" pod "3aa8284d-404b-4096-b89f-40a8cf493f51" (UID: "3aa8284d-404b-4096-b89f-40a8cf493f51"). InnerVolumeSpecName "kube-api-access-vrxmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:07.122022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.121997    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-xks-ca" pod "3aa8284d-404b-4096-b89f-40a8cf493f51" (UID: "3aa8284d-404b-4096-b89f-40a8cf493f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:07.221682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.221604    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa8284d-404b-4096-b89f-40a8cf493f51-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:07.221682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.221632    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxmh\" (UniqueName: \"kubernetes.io/projected/3aa8284d-404b-4096-b89f-40a8cf493f51-kube-api-access-vrxmh\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:07.907281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.907255    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"
Oct 29 23:31:07.907473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.907295    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j" event={"ID":"3aa8284d-404b-4096-b89f-40a8cf493f51","Type":"ContainerDied","Data":"c970d4392a01d29e75cb2aea793b9cf64a08ad77fb4da6ec4093e4d86f265f3c"}
Oct 29 23:31:07.907473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:07.907347    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c970d4392a01d29e75cb2aea793b9cf64a08ad77fb4da6ec4093e4d86f265f3c"
Oct 29 23:31:08.920441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:08.920410    4827 generic.go:356] "Generic (PLEG): container finished" podID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerID="7b1af9d45ddba4568f35fb8bbe7f5d458639ca64fedd0b2ad8768a7137b23636" exitCode=0
Oct 29 23:31:08.920966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:08.920483    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" event={"ID":"d06c4719-725f-4a22-8359-8d5d2de03a60","Type":"ContainerDied","Data":"7b1af9d45ddba4568f35fb8bbe7f5d458639ca64fedd0b2ad8768a7137b23636"}
Oct 29 23:31:08.920966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:08.920600    4827 scope.go:122] "RemoveContainer" containerID="e3a2070c02710eca274382963ceff2c14f999130a263c9d8b9f21ec14e8a7a81"
Oct 29 23:31:08.997287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:08.997265    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:31:09.051038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051004    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-credential-socket\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051071    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06c4719-725f-4a22-8359-8d5d2de03a60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d06c4719-725f-4a22-8359-8d5d2de03a60-istiod-ca-cert\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051134    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-podinfo\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051182    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-socket\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051212    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-token\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-token\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051328    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-data\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051386    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-envoy\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051424    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-certs\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.051671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.051541    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-kube-api-access-2ncvz\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-kube-api-access-2ncvz\") pod \"d06c4719-725f-4a22-8359-8d5d2de03a60\" (UID: \"d06c4719-725f-4a22-8359-8d5d2de03a60\") "
Oct 29 23:31:09.052650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.052587    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-socket" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:09.054407 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.053926    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06c4719-725f-4a22-8359-8d5d2de03a60-istiod-ca-cert" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:09.055261 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.055080    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-certs" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:09.055564 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.055532    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-data" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:09.055754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.055718    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-credential-socket" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:09.058315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.058288    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-kube-api-access-2ncvz" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "kube-api-access-2ncvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:09.058534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.058509    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-token" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:09.058797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.058759    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-podinfo" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:31:09.062099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.062071    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-envoy" pod "d06c4719-725f-4a22-8359-8d5d2de03a60" (UID: "d06c4719-725f-4a22-8359-8d5d2de03a60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:09.153538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153508    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153538 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153535    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153547    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153555    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153563    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153570    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153578    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ncvz\" (UniqueName: \"kubernetes.io/projected/d06c4719-725f-4a22-8359-8d5d2de03a60-kube-api-access-2ncvz\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153586    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d06c4719-725f-4a22-8359-8d5d2de03a60-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.153720 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.153595    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d06c4719-725f-4a22-8359-8d5d2de03a60-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:09.929801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.929748    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"
Oct 29 23:31:09.930287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.929806    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d" event={"ID":"d06c4719-725f-4a22-8359-8d5d2de03a60","Type":"ContainerDied","Data":"0fe74246e5af915168883461b37b8292c30ff8e70740d418e17322479488560d"}
Oct 29 23:31:09.930287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.929862    4827 scope.go:122] "RemoveContainer" containerID="7b1af9d45ddba4568f35fb8bbe7f5d458639ca64fedd0b2ad8768a7137b23636"
Oct 29 23:31:09.990813 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.990780    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"]
Oct 29 23:31:09.999858 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:09.999833    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-j795d"]
Oct 29 23:31:10.489577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:10.489547    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" path="/var/lib/kubelet/pods/d06c4719-725f-4a22-8359-8d5d2de03a60/volumes"
Oct 29 23:31:19.636399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:19.636366    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"]
Oct 29 23:31:19.636869 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:19.636580    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t" podUID="3d61656d-824f-4b95-b46e-7d7d4cec7645" containerName="maas-api" containerID="cri-o://36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64" gracePeriod=30
Oct 29 23:31:19.985451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:19.985427    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:31:20.004385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.004355    4827 generic.go:356] "Generic (PLEG): container finished" podID="3d61656d-824f-4b95-b46e-7d7d4cec7645" containerID="36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64" exitCode=0
Oct 29 23:31:20.004560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.004531    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t" event={"ID":"3d61656d-824f-4b95-b46e-7d7d4cec7645","Type":"ContainerDied","Data":"36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64"}
Oct 29 23:31:20.004635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.004585    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t" event={"ID":"3d61656d-824f-4b95-b46e-7d7d4cec7645","Type":"ContainerDied","Data":"c92b33bd05cb42d89a4c979abbe5fdc2f99e2a8ff82e8ed6d98d407ee2049a1c"}
Oct 29 23:31:20.004635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.004617    4827 scope.go:122] "RemoveContainer" containerID="36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64"
Oct 29 23:31:20.004751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.004681    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"
Oct 29 23:31:20.048497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.046711    4827 scope.go:122] "RemoveContainer" containerID="36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64"
Oct 29 23:31:20.048497 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:31:20.048165    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64\": container with ID starting with 36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64 not found: ID does not exist" containerID="36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64"
Oct 29 23:31:20.048497 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.048201    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64"} err="failed to get container status \"36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64\": rpc error: code = NotFound desc = could not find container \"36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64\": container with ID starting with 36bafa06c03b79848cd7de92f82d5e600e2dc53554e59fea79a313c4210e2e64 not found: ID does not exist"
Oct 29 23:31:20.136483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.136453    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d61656d-824f-4b95-b46e-7d7d4cec7645-kube-api-access-gwms6\" (UniqueName: \"kubernetes.io/projected/3d61656d-824f-4b95-b46e-7d7d4cec7645-kube-api-access-gwms6\") pod \"3d61656d-824f-4b95-b46e-7d7d4cec7645\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") "
Oct 29 23:31:20.136652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.136587    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-metrics-certs\") pod \"3d61656d-824f-4b95-b46e-7d7d4cec7645\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") "
Oct 29 23:31:20.136652 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.136643    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-maas-api-tls\") pod \"3d61656d-824f-4b95-b46e-7d7d4cec7645\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") "
Oct 29 23:31:20.136760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.136689    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d61656d-824f-4b95-b46e-7d7d4cec7645-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d61656d-824f-4b95-b46e-7d7d4cec7645-openshift-service-ca\") pod \"3d61656d-824f-4b95-b46e-7d7d4cec7645\" (UID: \"3d61656d-824f-4b95-b46e-7d7d4cec7645\") "
Oct 29 23:31:20.137214 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.137184    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d61656d-824f-4b95-b46e-7d7d4cec7645-openshift-service-ca" pod "3d61656d-824f-4b95-b46e-7d7d4cec7645" (UID: "3d61656d-824f-4b95-b46e-7d7d4cec7645"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:20.137563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.137541    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d61656d-824f-4b95-b46e-7d7d4cec7645-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:20.139326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.139302    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-metrics-certs" pod "3d61656d-824f-4b95-b46e-7d7d4cec7645" (UID: "3d61656d-824f-4b95-b46e-7d7d4cec7645"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:20.139435 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.139356    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-maas-api-tls" pod "3d61656d-824f-4b95-b46e-7d7d4cec7645" (UID: "3d61656d-824f-4b95-b46e-7d7d4cec7645"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:20.139626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.139594    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d61656d-824f-4b95-b46e-7d7d4cec7645-kube-api-access-gwms6" pod "3d61656d-824f-4b95-b46e-7d7d4cec7645" (UID: "3d61656d-824f-4b95-b46e-7d7d4cec7645"). InnerVolumeSpecName "kube-api-access-gwms6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:20.238588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.238506    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwms6\" (UniqueName: \"kubernetes.io/projected/3d61656d-824f-4b95-b46e-7d7d4cec7645-kube-api-access-gwms6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:20.238588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.238536    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:20.238588 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.238546    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d61656d-824f-4b95-b46e-7d7d4cec7645-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:20.354880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.354839    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"]
Oct 29 23:31:20.357981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.357955    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed3bf2-67d66789bc-bbj7t"]
Oct 29 23:31:20.366703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.366678    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"]
Oct 29 23:31:20.369288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.369268    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed3bf2-kkm5j"]
Oct 29 23:31:20.492036 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.491954    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa8284d-404b-4096-b89f-40a8cf493f51" path="/var/lib/kubelet/pods/3aa8284d-404b-4096-b89f-40a8cf493f51/volumes"
Oct 29 23:31:20.492552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:20.492534    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d61656d-824f-4b95-b46e-7d7d4cec7645" path="/var/lib/kubelet/pods/3d61656d-824f-4b95-b46e-7d7d4cec7645/volumes"
Oct 29 23:31:27.943438 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:27.943349    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"]
Oct 29 23:31:27.944236 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:27.944180    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" containerID="cri-o://8097a09fcc6ef7c0dadf35162c4f1855ae5f67f770693baf857297ba2d9ce7a6" gracePeriod=30
Oct 29 23:31:28.625749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.625716    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"]
Oct 29 23:31:28.626281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626265    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:28.626335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626281    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:28.626335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626306    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3d61656d-824f-4b95-b46e-7d7d4cec7645" containerName="maas-api"
Oct 29 23:31:28.626335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626311    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d61656d-824f-4b95-b46e-7d7d4cec7645" containerName="maas-api"
Oct 29 23:31:28.626335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626322    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy"
Oct 29 23:31:28.626335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626327    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy"
Oct 29 23:31:28.626335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626335    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3aa8284d-404b-4096-b89f-40a8cf493f51" containerName="revoke-keys"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626340    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa8284d-404b-4096-b89f-40a8cf493f51" containerName="revoke-keys"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626350    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626355    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626372    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626376    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626449    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="54dc02e0-b2a2-415b-8bfe-aefb65b3b68c" containerName="cleanup"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626460    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d61656d-824f-4b95-b46e-7d7d4cec7645" containerName="maas-api"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626470    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626476    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa8284d-404b-4096-b89f-40a8cf493f51" containerName="revoke-keys"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626483    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06c4719-725f-4a22-8359-8d5d2de03a60" containerName="istio-proxy"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626488    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4236ff5-dcd8-4ea8-b207-10491fa9ae6c" containerName="cleanup"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626493    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5f06e7-73db-48ae-987a-a62398a4b2b7" containerName="cleanup"
Oct 29 23:31:28.626537 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.626499    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb1c9e2-a882-4337-86d6-6cf164554657" containerName="cleanup"
Oct 29 23:31:28.631297 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.631275    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.651260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.651235    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"]
Oct 29 23:31:28.698211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.698175    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vzc\" (UniqueName: \"kubernetes.io/projected/a282c7fc-3a9c-4314-840c-a1edbf26f12f-kube-api-access-27vzc\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.698423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.698243    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.698504 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.698445    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.715675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.715638    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:28.715850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.715704    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:31:28.799690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.799659    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.799876 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.799825    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vzc\" (UniqueName: \"kubernetes.io/projected/a282c7fc-3a9c-4314-840c-a1edbf26f12f-kube-api-access-27vzc\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.799936 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.799878    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.800535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.800514    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.802172 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.802154    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.821442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.821415    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vzc\" (UniqueName: \"kubernetes.io/projected/a282c7fc-3a9c-4314-840c-a1edbf26f12f-kube-api-access-27vzc\") pod \"maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:28.944526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:28.944445    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:29.187444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:29.187406    4827 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:31:29.187661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:29.187470    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:31:29.248862 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:29.248829    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"]
Oct 29 23:31:29.250488 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:31:29.250460    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda282c7fc_3a9c_4314_840c_a1edbf26f12f.slice/crio-41c6ee945e7733dd68676229776f5967c227abd6186a6bfece1acedbf6363274 WatchSource:0}: Error finding container 41c6ee945e7733dd68676229776f5967c227abd6186a6bfece1acedbf6363274: Status 404 returned error can't find the container with id 41c6ee945e7733dd68676229776f5967c227abd6186a6bfece1acedbf6363274
Oct 29 23:31:30.097687 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:30.097650    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj" event={"ID":"a282c7fc-3a9c-4314-840c-a1edbf26f12f","Type":"ContainerStarted","Data":"fb5c44cae74e10c2c084612c8198584437641cbb26941bb26d9fe5a4944225a1"}
Oct 29 23:31:30.098177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:30.097709    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj" event={"ID":"a282c7fc-3a9c-4314-840c-a1edbf26f12f","Type":"ContainerStarted","Data":"41c6ee945e7733dd68676229776f5967c227abd6186a6bfece1acedbf6363274"}
Oct 29 23:31:30.136156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:30.136092    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj" podStartSLOduration=2.1360716809999998 podStartE2EDuration="2.136071681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:30.134294059 +0000 UTC m=+2925.757205081" watchObservedRunningTime="2026-10-29 23:31:30.136071681 +0000 UTC m=+2925.758982702"
Oct 29 23:31:31.105884 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:31.105836    4827 generic.go:356] "Generic (PLEG): container finished" podID="a282c7fc-3a9c-4314-840c-a1edbf26f12f" containerID="fb5c44cae74e10c2c084612c8198584437641cbb26941bb26d9fe5a4944225a1" exitCode=0
Oct 29 23:31:31.106299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:31.105923    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj" event={"ID":"a282c7fc-3a9c-4314-840c-a1edbf26f12f","Type":"ContainerDied","Data":"fb5c44cae74e10c2c084612c8198584437641cbb26941bb26d9fe5a4944225a1"}
Oct 29 23:31:32.326837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.326810    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:32.347330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.347296    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"]
Oct 29 23:31:32.347987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.347965    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a282c7fc-3a9c-4314-840c-a1edbf26f12f" containerName="revoke-keys"
Oct 29 23:31:32.347987 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.347986    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a282c7fc-3a9c-4314-840c-a1edbf26f12f" containerName="revoke-keys"
Oct 29 23:31:32.348174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.348130    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a282c7fc-3a9c-4314-840c-a1edbf26f12f" containerName="revoke-keys"
Oct 29 23:31:32.351566 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.351545    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.371466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.371408    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"]
Oct 29 23:31:32.444993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.444961    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a282c7fc-3a9c-4314-840c-a1edbf26f12f-kube-api-access-27vzc\" (UniqueName: \"kubernetes.io/projected/a282c7fc-3a9c-4314-840c-a1edbf26f12f-kube-api-access-27vzc\") pod \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") "
Oct 29 23:31:32.445160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.445098    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-openshift-ca\") pod \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") "
Oct 29 23:31:32.445160 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.445133    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-xks-ca\") pod \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\" (UID: \"a282c7fc-3a9c-4314-840c-a1edbf26f12f\") "
Oct 29 23:31:32.445397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.445374    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.445471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.445433    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.445471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.445448    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-openshift-ca" pod "a282c7fc-3a9c-4314-840c-a1edbf26f12f" (UID: "a282c7fc-3a9c-4314-840c-a1edbf26f12f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:32.445638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.445614    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqht6\" (UniqueName: \"kubernetes.io/projected/fb265e4f-895b-41d0-8898-e40f24e1ff10-kube-api-access-gqht6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.445746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.445732    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:32.447137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.447118    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-xks-ca" pod "a282c7fc-3a9c-4314-840c-a1edbf26f12f" (UID: "a282c7fc-3a9c-4314-840c-a1edbf26f12f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:32.447248 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.447228    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a282c7fc-3a9c-4314-840c-a1edbf26f12f-kube-api-access-27vzc" pod "a282c7fc-3a9c-4314-840c-a1edbf26f12f" (UID: "a282c7fc-3a9c-4314-840c-a1edbf26f12f"). InnerVolumeSpecName "kube-api-access-27vzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:32.547056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.547023    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqht6\" (UniqueName: \"kubernetes.io/projected/fb265e4f-895b-41d0-8898-e40f24e1ff10-kube-api-access-gqht6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.547243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.547202    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.547319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.547253    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.547385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.547373    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vzc\" (UniqueName: \"kubernetes.io/projected/a282c7fc-3a9c-4314-840c-a1edbf26f12f-kube-api-access-27vzc\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:32.547460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.547390    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282c7fc-3a9c-4314-840c-a1edbf26f12f-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:32.548018 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.547997    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.549572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.549554    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.568452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.568427    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqht6\" (UniqueName: \"kubernetes.io/projected/fb265e4f-895b-41d0-8898-e40f24e1ff10-kube-api-access-gqht6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.666527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.666478    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:32.827838 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:31:32.827807    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:31:32.989198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:32.989167    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"]
Oct 29 23:31:33.019262 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:31:33.019228    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb265e4f_895b_41d0_8898_e40f24e1ff10.slice/crio-87cc41ab12204204ec0781ecc914db346193d82c5f5ff98c4321f83242d3b8a1 WatchSource:0}: Error finding container 87cc41ab12204204ec0781ecc914db346193d82c5f5ff98c4321f83242d3b8a1: Status 404 returned error can't find the container with id 87cc41ab12204204ec0781ecc914db346193d82c5f5ff98c4321f83242d3b8a1
Oct 29 23:31:33.122013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.121982    4827 generic.go:356] "Generic (PLEG): container finished" podID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerID="8097a09fcc6ef7c0dadf35162c4f1855ae5f67f770693baf857297ba2d9ce7a6" exitCode=0
Oct 29 23:31:33.122131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.122049    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" event={"ID":"8ea8da08-ef56-4327-becb-86c5dff41b70","Type":"ContainerDied","Data":"8097a09fcc6ef7c0dadf35162c4f1855ae5f67f770693baf857297ba2d9ce7a6"}
Oct 29 23:31:33.122131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.122111    4827 scope.go:122] "RemoveContainer" containerID="722f3586bcc8062d906c98c8c71c5f4ec14d332cbdc8669e7781ff552f5021ca"
Oct 29 23:31:33.124257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.124235    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"
Oct 29 23:31:33.124378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.124271    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj" event={"ID":"a282c7fc-3a9c-4314-840c-a1edbf26f12f","Type":"ContainerDied","Data":"41c6ee945e7733dd68676229776f5967c227abd6186a6bfece1acedbf6363274"}
Oct 29 23:31:33.124378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.124315    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c6ee945e7733dd68676229776f5967c227abd6186a6bfece1acedbf6363274"
Oct 29 23:31:33.125982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.125959    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2" event={"ID":"fb265e4f-895b-41d0-8898-e40f24e1ff10","Type":"ContainerStarted","Data":"87cc41ab12204204ec0781ecc914db346193d82c5f5ff98c4321f83242d3b8a1"}
Oct 29 23:31:33.203801 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.203759    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:31:33.255185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255080    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-kube-api-access-62qwl\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-kube-api-access-62qwl\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255144    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-podinfo\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255222    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-socket\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255263    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-token\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255328    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-credential-socket\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255456 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255355    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea8da08-ef56-4327-becb-86c5dff41b70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea8da08-ef56-4327-becb-86c5dff41b70-istiod-ca-cert\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255520    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-socket" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:33.255698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255661    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-envoy\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255672    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-credential-socket" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:33.255891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255728    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-certs\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.255891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255744    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea8da08-ef56-4327-becb-86c5dff41b70-istiod-ca-cert" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:33.255891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.255788    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-data\") pod \"8ea8da08-ef56-4327-becb-86c5dff41b70\" (UID: \"8ea8da08-ef56-4327-becb-86c5dff41b70\") "
Oct 29 23:31:33.256081 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.256059    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-certs" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:33.256144 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.256072    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-data" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:33.256385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.256367    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.256465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.256389    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea8da08-ef56-4327-becb-86c5dff41b70-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.256465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.256403    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.256465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.256415    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.256465 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.256429    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.258174 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.258144    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-envoy" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:33.258288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.258187    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-token" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:33.258288 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.258228    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-podinfo" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:31:33.259276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.259257    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-kube-api-access-62qwl" pod "8ea8da08-ef56-4327-becb-86c5dff41b70" (UID: "8ea8da08-ef56-4327-becb-86c5dff41b70"). InnerVolumeSpecName "kube-api-access-62qwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:33.357547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.357516    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.357547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.357544    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62qwl\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-kube-api-access-62qwl\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.357547 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.357554    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.358027 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.357563    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea8da08-ef56-4327-becb-86c5dff41b70-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:33.379570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.379497    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"]
Oct 29 23:31:33.390292 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.390261    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"]
Oct 29 23:31:33.396368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:33.396343    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"]
Oct 29 23:31:34.134981 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.134943    4827 generic.go:356] "Generic (PLEG): container finished" podID="fb265e4f-895b-41d0-8898-e40f24e1ff10" containerID="e098189c15a1d8fa06a88c531506815744caf39be8c36c0bcd39913fcb7afe0f" exitCode=0
Oct 29 23:31:34.135169 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.135019    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2" event={"ID":"fb265e4f-895b-41d0-8898-e40f24e1ff10","Type":"ContainerDied","Data":"e098189c15a1d8fa06a88c531506815744caf39be8c36c0bcd39913fcb7afe0f"}
Oct 29 23:31:34.136634 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.136615    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"
Oct 29 23:31:34.136754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.136616    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67" event={"ID":"8ea8da08-ef56-4327-becb-86c5dff41b70","Type":"ContainerDied","Data":"0257317de01450eace9696e1c06676b21a8cd056b786cc0d2988f874c9aed309"}
Oct 29 23:31:34.136754 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.136728    4827 scope.go:122] "RemoveContainer" containerID="8097a09fcc6ef7c0dadf35162c4f1855ae5f67f770693baf857297ba2d9ce7a6"
Oct 29 23:31:34.207007 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.206975    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"]
Oct 29 23:31:34.212463 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.212439    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-48t67"]
Oct 29 23:31:34.487838 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:34.487810    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" path="/var/lib/kubelet/pods/8ea8da08-ef56-4327-becb-86c5dff41b70/volumes"
Oct 29 23:31:35.292068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.292046    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:35.383341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.383302    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb265e4f-895b-41d0-8898-e40f24e1ff10-kube-api-access-gqht6\" (UniqueName: \"kubernetes.io/projected/fb265e4f-895b-41d0-8898-e40f24e1ff10-kube-api-access-gqht6\") pod \"fb265e4f-895b-41d0-8898-e40f24e1ff10\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") "
Oct 29 23:31:35.383565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.383549    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-openshift-ca\") pod \"fb265e4f-895b-41d0-8898-e40f24e1ff10\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") "
Oct 29 23:31:35.383665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.383621    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-xks-ca\") pod \"fb265e4f-895b-41d0-8898-e40f24e1ff10\" (UID: \"fb265e4f-895b-41d0-8898-e40f24e1ff10\") "
Oct 29 23:31:35.383958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.383933    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-openshift-ca" pod "fb265e4f-895b-41d0-8898-e40f24e1ff10" (UID: "fb265e4f-895b-41d0-8898-e40f24e1ff10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:35.384100 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.384083    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:35.385451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.385431    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb265e4f-895b-41d0-8898-e40f24e1ff10-kube-api-access-gqht6" pod "fb265e4f-895b-41d0-8898-e40f24e1ff10" (UID: "fb265e4f-895b-41d0-8898-e40f24e1ff10"). InnerVolumeSpecName "kube-api-access-gqht6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:35.385680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.385661    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-xks-ca" pod "fb265e4f-895b-41d0-8898-e40f24e1ff10" (UID: "fb265e4f-895b-41d0-8898-e40f24e1ff10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:35.488210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.488062    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqht6\" (UniqueName: \"kubernetes.io/projected/fb265e4f-895b-41d0-8898-e40f24e1ff10-kube-api-access-gqht6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:35.488210 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:35.488175    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb265e4f-895b-41d0-8898-e40f24e1ff10-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:36.155099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:36.155065    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"
Oct 29 23:31:36.155262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:36.155094    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2" event={"ID":"fb265e4f-895b-41d0-8898-e40f24e1ff10","Type":"ContainerDied","Data":"87cc41ab12204204ec0781ecc914db346193d82c5f5ff98c4321f83242d3b8a1"}
Oct 29 23:31:36.155262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:36.155149    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87cc41ab12204204ec0781ecc914db346193d82c5f5ff98c4321f83242d3b8a1"
Oct 29 23:31:39.123434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.122416    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"]
Oct 29 23:31:39.890854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.890814    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"]
Oct 29 23:31:39.892994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.892962    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy"
Oct 29 23:31:39.892994 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.892993    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy"
Oct 29 23:31:39.893206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.893026    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fb265e4f-895b-41d0-8898-e40f24e1ff10" containerName="revoke-keys"
Oct 29 23:31:39.893206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.893036    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb265e4f-895b-41d0-8898-e40f24e1ff10" containerName="revoke-keys"
Oct 29 23:31:39.893206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.893057    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy"
Oct 29 23:31:39.893206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.893064    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy"
Oct 29 23:31:39.919260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.911902    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy"
Oct 29 23:31:39.919260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.911974    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb265e4f-895b-41d0-8898-e40f24e1ff10" containerName="revoke-keys"
Oct 29 23:31:39.966014 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.965985    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"]
Oct 29 23:31:39.966365 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.966303    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:39.990340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.990315    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:39.990472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:39.990455    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:31:40.011802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.011730    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.011988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.011887    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf46w\" (UniqueName: \"kubernetes.io/projected/d9657ce0-51eb-46f9-a701-645f98c1b6f2-kube-api-access-pf46w\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.011988 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.011954    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.113391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.113346    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.113587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.113445    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf46w\" (UniqueName: \"kubernetes.io/projected/d9657ce0-51eb-46f9-a701-645f98c1b6f2-kube-api-access-pf46w\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.113587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.113521    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.114335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.114307    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.115731 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.115705    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.134944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.134920    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf46w\" (UniqueName: \"kubernetes.io/projected/d9657ce0-51eb-46f9-a701-645f98c1b6f2-kube-api-access-pf46w\") pod \"maas-api-revoke-sub-f6b0afa6-58j7w\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.192048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.191970    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"]
Oct 29 23:31:40.192956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.192934    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea8da08-ef56-4327-becb-86c5dff41b70" containerName="istio-proxy"
Oct 29 23:31:40.197035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.197014    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.214861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.214834    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.215035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.214870    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkll6\" (UniqueName: \"kubernetes.io/projected/0858e0f5-5c00-43cd-8cd4-582d03797945-kube-api-access-mkll6\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.215120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.215030    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.216643 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.216617    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"]
Oct 29 23:31:40.298904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.290017    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:40.319458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.319411    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.319649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.319570    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.319649 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.319602    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkll6\" (UniqueName: \"kubernetes.io/projected/0858e0f5-5c00-43cd-8cd4-582d03797945-kube-api-access-mkll6\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.320607 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.320556    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.337909 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.337877    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.342325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.342299    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkll6\" (UniqueName: \"kubernetes.io/projected/0858e0f5-5c00-43cd-8cd4-582d03797945-kube-api-access-mkll6\") pod \"maas-api-revoke-sub-e7c70e40-dlncv\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.517997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.517965    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:40.762131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.761630    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"]
Oct 29 23:31:40.988866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:40.988820    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"]
Oct 29 23:31:41.193154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:41.193117    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w" event={"ID":"d9657ce0-51eb-46f9-a701-645f98c1b6f2","Type":"ContainerStarted","Data":"0a50eaf1678872a1628f8433d492720aeebf39d0f0daba0d58d03233afed239d"}
Oct 29 23:31:41.195405 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:41.195377    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv" event={"ID":"0858e0f5-5c00-43cd-8cd4-582d03797945","Type":"ContainerStarted","Data":"af063b1c50fb699d3118fa870a67d6df4ebf2a6a432fc83cbe0074dd37fd0e57"}
Oct 29 23:31:42.203846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:42.203807    4827 generic.go:356] "Generic (PLEG): container finished" podID="d9657ce0-51eb-46f9-a701-645f98c1b6f2" containerID="c9910c2231737c0f042f828a0e3204fcf0d7cc513c531efe4c40628afc994b90" exitCode=0
Oct 29 23:31:42.204303 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:42.203886    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w" event={"ID":"d9657ce0-51eb-46f9-a701-645f98c1b6f2","Type":"ContainerDied","Data":"c9910c2231737c0f042f828a0e3204fcf0d7cc513c531efe4c40628afc994b90"}
Oct 29 23:31:42.205674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:42.205639    4827 generic.go:356] "Generic (PLEG): container finished" podID="0858e0f5-5c00-43cd-8cd4-582d03797945" containerID="40d4960ed1d0ff66be859596408e56601bf02844775b859e38852a743e90da7d" exitCode=0
Oct 29 23:31:42.205810 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:42.205699    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv" event={"ID":"0858e0f5-5c00-43cd-8cd4-582d03797945","Type":"ContainerDied","Data":"40d4960ed1d0ff66be859596408e56601bf02844775b859e38852a743e90da7d"}
Oct 29 23:31:43.363110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.363086    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:43.393368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.393344    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-openshift-ca\") pod \"0858e0f5-5c00-43cd-8cd4-582d03797945\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") "
Oct 29 23:31:43.393494 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.393397    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0858e0f5-5c00-43cd-8cd4-582d03797945-kube-api-access-mkll6\" (UniqueName: \"kubernetes.io/projected/0858e0f5-5c00-43cd-8cd4-582d03797945-kube-api-access-mkll6\") pod \"0858e0f5-5c00-43cd-8cd4-582d03797945\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") "
Oct 29 23:31:43.393665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.393644    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-openshift-ca" pod "0858e0f5-5c00-43cd-8cd4-582d03797945" (UID: "0858e0f5-5c00-43cd-8cd4-582d03797945"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:43.393790 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.393761    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:43.395489 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.395466    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0858e0f5-5c00-43cd-8cd4-582d03797945-kube-api-access-mkll6" pod "0858e0f5-5c00-43cd-8cd4-582d03797945" (UID: "0858e0f5-5c00-43cd-8cd4-582d03797945"). InnerVolumeSpecName "kube-api-access-mkll6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:43.414599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.414560    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:43.494839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.494508    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-xks-ca\") pod \"0858e0f5-5c00-43cd-8cd4-582d03797945\" (UID: \"0858e0f5-5c00-43cd-8cd4-582d03797945\") "
Oct 29 23:31:43.503822 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.495055    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkll6\" (UniqueName: \"kubernetes.io/projected/0858e0f5-5c00-43cd-8cd4-582d03797945-kube-api-access-mkll6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:43.521942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.521897    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-xks-ca" pod "0858e0f5-5c00-43cd-8cd4-582d03797945" (UID: "0858e0f5-5c00-43cd-8cd4-582d03797945"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:43.596355 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.596319    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-openshift-ca\") pod \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") "
Oct 29 23:31:43.596534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.596409    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9657ce0-51eb-46f9-a701-645f98c1b6f2-kube-api-access-pf46w\" (UniqueName: \"kubernetes.io/projected/d9657ce0-51eb-46f9-a701-645f98c1b6f2-kube-api-access-pf46w\") pod \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") "
Oct 29 23:31:43.596534 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.596488    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-xks-ca\") pod \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\" (UID: \"d9657ce0-51eb-46f9-a701-645f98c1b6f2\") "
Oct 29 23:31:43.596789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.596739    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-openshift-ca" pod "d9657ce0-51eb-46f9-a701-645f98c1b6f2" (UID: "d9657ce0-51eb-46f9-a701-645f98c1b6f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:43.597182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.597131    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:43.597182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.597155    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858e0f5-5c00-43cd-8cd4-582d03797945-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:43.598593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.598572    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-xks-ca" pod "d9657ce0-51eb-46f9-a701-645f98c1b6f2" (UID: "d9657ce0-51eb-46f9-a701-645f98c1b6f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:43.598664 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.598606    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9657ce0-51eb-46f9-a701-645f98c1b6f2-kube-api-access-pf46w" pod "d9657ce0-51eb-46f9-a701-645f98c1b6f2" (UID: "d9657ce0-51eb-46f9-a701-645f98c1b6f2"). InnerVolumeSpecName "kube-api-access-pf46w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:43.698033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.698000    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf46w\" (UniqueName: \"kubernetes.io/projected/d9657ce0-51eb-46f9-a701-645f98c1b6f2-kube-api-access-pf46w\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:43.698033 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:43.698039    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9657ce0-51eb-46f9-a701-645f98c1b6f2-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:44.175803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:44.175716    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" podUID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" containerName="payload-pre-processing" containerID="cri-o://cc8189589deaba35c57b7cd4a97b7927c789cacec26685fae8ad10c8488a30e4" gracePeriod=25
Oct 29 23:31:44.220541 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:44.220510    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w"
Oct 29 23:31:44.220700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:44.220539    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b0afa6-58j7w" event={"ID":"d9657ce0-51eb-46f9-a701-645f98c1b6f2","Type":"ContainerDied","Data":"0a50eaf1678872a1628f8433d492720aeebf39d0f0daba0d58d03233afed239d"}
Oct 29 23:31:44.220700 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:44.220634    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a50eaf1678872a1628f8433d492720aeebf39d0f0daba0d58d03233afed239d"
Oct 29 23:31:44.221963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:44.221947    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv"
Oct 29 23:31:44.222094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:44.221995    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c70e40-dlncv" event={"ID":"0858e0f5-5c00-43cd-8cd4-582d03797945","Type":"ContainerDied","Data":"af063b1c50fb699d3118fa870a67d6df4ebf2a6a432fc83cbe0074dd37fd0e57"}
Oct 29 23:31:44.222094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:44.222045    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af063b1c50fb699d3118fa870a67d6df4ebf2a6a432fc83cbe0074dd37fd0e57"
Oct 29 23:31:47.866460 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.866422    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"]
Oct 29 23:31:47.867249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.867223    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d9657ce0-51eb-46f9-a701-645f98c1b6f2" containerName="revoke-keys"
Oct 29 23:31:47.867249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.867248    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9657ce0-51eb-46f9-a701-645f98c1b6f2" containerName="revoke-keys"
Oct 29 23:31:47.867249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.867262    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0858e0f5-5c00-43cd-8cd4-582d03797945" containerName="revoke-keys"
Oct 29 23:31:47.867483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.867269    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0858e0f5-5c00-43cd-8cd4-582d03797945" containerName="revoke-keys"
Oct 29 23:31:47.867483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.867413    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9657ce0-51eb-46f9-a701-645f98c1b6f2" containerName="revoke-keys"
Oct 29 23:31:47.867483 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.867424    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0858e0f5-5c00-43cd-8cd4-582d03797945" containerName="revoke-keys"
Oct 29 23:31:47.871220 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.871201    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:47.884979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.884958    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:47.885103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.885082    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:31:47.888625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.888590    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"]
Oct 29 23:31:47.921003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.920968    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" podUID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:31:47.958119 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.958089    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:47.958283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.958145    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnx4s\" (UniqueName: \"kubernetes.io/projected/04413b7d-e0ce-42b6-874c-fc5152590b95-kube-api-access-qnx4s\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:47.958283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:47.958255    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.059625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.059568    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.059815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.059713    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.059815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.059744    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnx4s\" (UniqueName: \"kubernetes.io/projected/04413b7d-e0ce-42b6-874c-fc5152590b95-kube-api-access-qnx4s\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.060333 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.060308    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.061914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.061889    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.082032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.082005    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnx4s\" (UniqueName: \"kubernetes.io/projected/04413b7d-e0ce-42b6-874c-fc5152590b95-kube-api-access-qnx4s\") pod \"maas-api-revoke-sub-ba0a03df-b24f8\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.189492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.189452    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:48.401704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.401668    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" containerID="cri-o://174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd" gracePeriod=45
Oct 29 23:31:48.415249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.415177    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" containerID="cri-o://507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649" gracePeriod=45
Oct 29 23:31:48.416324 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.416296    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" containerID="cri-o://70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2" gracePeriod=45
Oct 29 23:31:48.502215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.502188    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"]
Oct 29 23:31:48.530088 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:31:48.530053    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04413b7d_e0ce_42b6_874c_fc5152590b95.slice/crio-cfbfb0984cd1d2bae8c089bb380ee7ec259a073ecace8be28c4deaf02eedbeac WatchSource:0}: Error finding container cfbfb0984cd1d2bae8c089bb380ee7ec259a073ecace8be28c4deaf02eedbeac: Status 404 returned error can't find the container with id cfbfb0984cd1d2bae8c089bb380ee7ec259a073ecace8be28c4deaf02eedbeac
Oct 29 23:31:48.575281 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.575094    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 29 23:31:48.710444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.710372    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:31:48.728963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.728880    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:31:48.735934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.735912    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:31:48.771749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771713    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-home\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-home\") pod \"a1833b55-765c-43e9-b608-a1c9ca16ca37\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") "
Oct 29 23:31:48.771955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771757    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tls-certs\") pod \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") "
Oct 29 23:31:48.771955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771806    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tmp-dir\") pod \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") "
Oct 29 23:31:48.771955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771831    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-model-cache\") pod \"2081f6b2-15ff-4280-9d56-917613a6af5e\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") "
Oct 29 23:31:48.771955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771863    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-dshm\") pod \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") "
Oct 29 23:31:48.771955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771888    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-model-cache\") pod \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") "
Oct 29 23:31:48.771955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771946    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1833b55-765c-43e9-b608-a1c9ca16ca37-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1833b55-765c-43e9-b608-a1c9ca16ca37-tls-certs\") pod \"a1833b55-765c-43e9-b608-a1c9ca16ca37\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.771984    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2081f6b2-15ff-4280-9d56-917613a6af5e-kube-api-access-hm8z7\" (UniqueName: \"kubernetes.io/projected/2081f6b2-15ff-4280-9d56-917613a6af5e-kube-api-access-hm8z7\") pod \"2081f6b2-15ff-4280-9d56-917613a6af5e\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772006    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-dshm\") pod \"a1833b55-765c-43e9-b608-a1c9ca16ca37\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772027    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-home\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-home\") pod \"2081f6b2-15ff-4280-9d56-917613a6af5e\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772027    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-home" pod "a1833b55-765c-43e9-b608-a1c9ca16ca37" (UID: "a1833b55-765c-43e9-b608-a1c9ca16ca37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772058    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-dshm\") pod \"2081f6b2-15ff-4280-9d56-917613a6af5e\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772078    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1833b55-765c-43e9-b608-a1c9ca16ca37-kube-api-access-j5nsz\" (UniqueName: \"kubernetes.io/projected/a1833b55-765c-43e9-b608-a1c9ca16ca37-kube-api-access-j5nsz\") pod \"a1833b55-765c-43e9-b608-a1c9ca16ca37\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772107    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-tmp-dir\") pod \"2081f6b2-15ff-4280-9d56-917613a6af5e\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772139    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-home\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-home\") pod \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772156    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-model-cache\") pod \"a1833b55-765c-43e9-b608-a1c9ca16ca37\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772206    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2081f6b2-15ff-4280-9d56-917613a6af5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2081f6b2-15ff-4280-9d56-917613a6af5e-tls-certs\") pod \"2081f6b2-15ff-4280-9d56-917613a6af5e\" (UID: \"2081f6b2-15ff-4280-9d56-917613a6af5e\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772206    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-model-cache" pod "2081f6b2-15ff-4280-9d56-917613a6af5e" (UID: "2081f6b2-15ff-4280-9d56-917613a6af5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772241    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-tmp-dir\") pod \"a1833b55-765c-43e9-b608-a1c9ca16ca37\" (UID: \"a1833b55-765c-43e9-b608-a1c9ca16ca37\") "
Oct 29 23:31:48.772373 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772266    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5e2cbd-6e8f-47e0-a282-743b6525c169-kube-api-access-xn7j9\" (UniqueName: \"kubernetes.io/projected/8f5e2cbd-6e8f-47e0-a282-743b6525c169-kube-api-access-xn7j9\") pod \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\" (UID: \"8f5e2cbd-6e8f-47e0-a282-743b6525c169\") "
Oct 29 23:31:48.773562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772719    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.773562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.772738    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.773802 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.773759    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tmp-dir" pod "8f5e2cbd-6e8f-47e0-a282-743b6525c169" (UID: "8f5e2cbd-6e8f-47e0-a282-743b6525c169"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.774362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.774342    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-model-cache" pod "8f5e2cbd-6e8f-47e0-a282-743b6525c169" (UID: "8f5e2cbd-6e8f-47e0-a282-743b6525c169"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.774683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.774665    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-home" pod "8f5e2cbd-6e8f-47e0-a282-743b6525c169" (UID: "8f5e2cbd-6e8f-47e0-a282-743b6525c169"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.775284    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-dshm" pod "8f5e2cbd-6e8f-47e0-a282-743b6525c169" (UID: "8f5e2cbd-6e8f-47e0-a282-743b6525c169"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.775357    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tls-certs" pod "8f5e2cbd-6e8f-47e0-a282-743b6525c169" (UID: "8f5e2cbd-6e8f-47e0-a282-743b6525c169"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.775615    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-home" pod "2081f6b2-15ff-4280-9d56-917613a6af5e" (UID: "2081f6b2-15ff-4280-9d56-917613a6af5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.775627    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-tmp-dir" pod "2081f6b2-15ff-4280-9d56-917613a6af5e" (UID: "2081f6b2-15ff-4280-9d56-917613a6af5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.775906    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-tmp-dir" pod "a1833b55-765c-43e9-b608-a1c9ca16ca37" (UID: "a1833b55-765c-43e9-b608-a1c9ca16ca37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.775933    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5e2cbd-6e8f-47e0-a282-743b6525c169-kube-api-access-xn7j9" pod "8f5e2cbd-6e8f-47e0-a282-743b6525c169" (UID: "8f5e2cbd-6e8f-47e0-a282-743b6525c169"). InnerVolumeSpecName "kube-api-access-xn7j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.776185    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-model-cache" pod "a1833b55-765c-43e9-b608-a1c9ca16ca37" (UID: "a1833b55-765c-43e9-b608-a1c9ca16ca37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.776922    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-dshm" pod "a1833b55-765c-43e9-b608-a1c9ca16ca37" (UID: "a1833b55-765c-43e9-b608-a1c9ca16ca37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.777780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.777141    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1833b55-765c-43e9-b608-a1c9ca16ca37-kube-api-access-j5nsz" pod "a1833b55-765c-43e9-b608-a1c9ca16ca37" (UID: "a1833b55-765c-43e9-b608-a1c9ca16ca37"). InnerVolumeSpecName "kube-api-access-j5nsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:48.778444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.778411    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-dshm" pod "2081f6b2-15ff-4280-9d56-917613a6af5e" (UID: "2081f6b2-15ff-4280-9d56-917613a6af5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:31:48.778897 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.778874    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2081f6b2-15ff-4280-9d56-917613a6af5e-tls-certs" pod "2081f6b2-15ff-4280-9d56-917613a6af5e" (UID: "2081f6b2-15ff-4280-9d56-917613a6af5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:48.778997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.778890    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1833b55-765c-43e9-b608-a1c9ca16ca37-tls-certs" pod "a1833b55-765c-43e9-b608-a1c9ca16ca37" (UID: "a1833b55-765c-43e9-b608-a1c9ca16ca37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:48.778997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.778962    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2081f6b2-15ff-4280-9d56-917613a6af5e-kube-api-access-hm8z7" pod "2081f6b2-15ff-4280-9d56-917613a6af5e" (UID: "2081f6b2-15ff-4280-9d56-917613a6af5e"). InnerVolumeSpecName "kube-api-access-hm8z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:48.873896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873868    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2081f6b2-15ff-4280-9d56-917613a6af5e-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.873896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873895    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873905    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn7j9\" (UniqueName: \"kubernetes.io/projected/8f5e2cbd-6e8f-47e0-a282-743b6525c169-kube-api-access-xn7j9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873914    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873922    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873930    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873938    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873948    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1833b55-765c-43e9-b608-a1c9ca16ca37-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873956    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm8z7\" (UniqueName: \"kubernetes.io/projected/2081f6b2-15ff-4280-9d56-917613a6af5e-kube-api-access-hm8z7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873964    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873973    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873980    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873988    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5nsz\" (UniqueName: \"kubernetes.io/projected/a1833b55-765c-43e9-b608-a1c9ca16ca37-kube-api-access-j5nsz\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.873996    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2081f6b2-15ff-4280-9d56-917613a6af5e-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.874004    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5e2cbd-6e8f-47e0-a282-743b6525c169-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:48.874421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:48.874012    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1833b55-765c-43e9-b608-a1c9ca16ca37-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:49.259753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.259663    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8" event={"ID":"04413b7d-e0ce-42b6-874c-fc5152590b95","Type":"ContainerStarted","Data":"150ccef7a3d3e2696e7ead8155e3f4d8ad4fd62fea8aed5b495f8e59cb991096"}
Oct 29 23:31:49.259753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.259730    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8" event={"ID":"04413b7d-e0ce-42b6-874c-fc5152590b95","Type":"ContainerStarted","Data":"cfbfb0984cd1d2bae8c089bb380ee7ec259a073ecace8be28c4deaf02eedbeac"}
Oct 29 23:31:49.261130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.261105    4827 generic.go:356] "Generic (PLEG): container finished" podID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerID="174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd" exitCode=0
Oct 29 23:31:49.261285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.261161    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"
Oct 29 23:31:49.261285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.261190    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" event={"ID":"a1833b55-765c-43e9-b608-a1c9ca16ca37","Type":"ContainerDied","Data":"174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd"}
Oct 29 23:31:49.261285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.261222    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" event={"ID":"a1833b55-765c-43e9-b608-a1c9ca16ca37","Type":"ContainerDied","Data":"bc6d93704bb3dea4a9797baeaad5e0f9ac0e1390491715fd72b5ade06c80f2c8"}
Oct 29 23:31:49.261285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.261240    4827 scope.go:122] "RemoveContainer" containerID="174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd"
Oct 29 23:31:49.262593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.262561    4827 generic.go:356] "Generic (PLEG): container finished" podID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerID="507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649" exitCode=0
Oct 29 23:31:49.262666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.262598    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"
Oct 29 23:31:49.262666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.262652    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" event={"ID":"2081f6b2-15ff-4280-9d56-917613a6af5e","Type":"ContainerDied","Data":"507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649"}
Oct 29 23:31:49.262799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.262678    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s" event={"ID":"2081f6b2-15ff-4280-9d56-917613a6af5e","Type":"ContainerDied","Data":"0d5989529f400825ee3df154c95c9f87c548242696359f36faf392b5ed915e07"}
Oct 29 23:31:49.264255 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.264233    4827 generic.go:356] "Generic (PLEG): container finished" podID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerID="70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2" exitCode=0
Oct 29 23:31:49.264372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.264274    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" event={"ID":"8f5e2cbd-6e8f-47e0-a282-743b6525c169","Type":"ContainerDied","Data":"70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2"}
Oct 29 23:31:49.264372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.264289    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"
Oct 29 23:31:49.264372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.264299    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" event={"ID":"8f5e2cbd-6e8f-47e0-a282-743b6525c169","Type":"ContainerDied","Data":"8de9d4b45c117c89bd01ce6c33ab0971460cb1e0094c91d586ea7a186ea8e005"}
Oct 29 23:31:49.274969 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.274953    4827 scope.go:122] "RemoveContainer" containerID="174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd"
Oct 29 23:31:49.275221 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:31:49.275202    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd\": container with ID starting with 174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd not found: ID does not exist" containerID="174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd"
Oct 29 23:31:49.275296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.275232    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd"} err="failed to get container status \"174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd\": rpc error: code = NotFound desc = could not find container \"174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd\": container with ID starting with 174f139cf6af10394e6cd71efdfad036e144165821ccecd384deaa774efdc6fd not found: ID does not exist"
Oct 29 23:31:49.275296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.275251    4827 scope.go:122] "RemoveContainer" containerID="507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649"
Oct 29 23:31:49.284257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.284239    4827 scope.go:122] "RemoveContainer" containerID="507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649"
Oct 29 23:31:49.284523 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:31:49.284503    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649\": container with ID starting with 507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649 not found: ID does not exist" containerID="507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649"
Oct 29 23:31:49.284591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.284529    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649"} err="failed to get container status \"507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649\": rpc error: code = NotFound desc = could not find container \"507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649\": container with ID starting with 507c9d365bd530a4310d6224842e63f5be07a8db6fb47bea59534aaad82b7649 not found: ID does not exist"
Oct 29 23:31:49.284591 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.284544    4827 scope.go:122] "RemoveContainer" containerID="70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2"
Oct 29 23:31:49.293110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.293093    4827 scope.go:122] "RemoveContainer" containerID="70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2"
Oct 29 23:31:49.293337 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:31:49.293318    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2\": container with ID starting with 70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2 not found: ID does not exist" containerID="70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2"
Oct 29 23:31:49.293400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.293347    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2"} err="failed to get container status \"70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2\": rpc error: code = NotFound desc = could not find container \"70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2\": container with ID starting with 70ec8bf39a0fee28a4a56de1f8428ba081c431ea950d1e0c3a963c87a42b85b2 not found: ID does not exist"
Oct 29 23:31:49.298966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.298924    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8" podStartSLOduration=2.298912055 podStartE2EDuration="2.298912055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:49.296103987 +0000 UTC m=+2944.919015008" watchObservedRunningTime="2026-10-29 23:31:49.298912055 +0000 UTC m=+2944.921823075"
Oct 29 23:31:49.330064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.330046    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"]
Oct 29 23:31:49.332891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.332870    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2"]
Oct 29 23:31:49.363553 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.363532    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"]
Oct 29 23:31:49.366163 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.366140    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p"]
Oct 29 23:31:49.396362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.396339    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"]
Oct 29 23:31:49.399084 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.399051    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/premium-w2-575d59-kserve-7bdbc84585-d9d7s"]
Oct 29 23:31:49.618352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.618320    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/simulator-w2-575d59-kserve-856dbcbdfb-kdc2p" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:49.643101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:49.643066    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-w2-575d59-kserve-678d58cc44-ztkj2" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:50.272245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:50.272209    4827 generic.go:356] "Generic (PLEG): container finished" podID="04413b7d-e0ce-42b6-874c-fc5152590b95" containerID="150ccef7a3d3e2696e7ead8155e3f4d8ad4fd62fea8aed5b495f8e59cb991096" exitCode=0
Oct 29 23:31:50.272671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:50.272295    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8" event={"ID":"04413b7d-e0ce-42b6-874c-fc5152590b95","Type":"ContainerDied","Data":"150ccef7a3d3e2696e7ead8155e3f4d8ad4fd62fea8aed5b495f8e59cb991096"}
Oct 29 23:31:50.488581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:50.488552    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" path="/var/lib/kubelet/pods/2081f6b2-15ff-4280-9d56-917613a6af5e/volumes"
Oct 29 23:31:50.489050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:50.489035    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" path="/var/lib/kubelet/pods/8f5e2cbd-6e8f-47e0-a282-743b6525c169/volumes"
Oct 29 23:31:50.489444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:50.489432    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" path="/var/lib/kubelet/pods/a1833b55-765c-43e9-b608-a1c9ca16ca37/volumes"
Oct 29 23:31:51.421912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.421885    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:51.484015 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.483983    4827 scope.go:122] "RemoveContainer" containerID="98d358d6efba0cadca615c4cf31e429dead8feb7a95128e2bd0baf12ead6df3a"
Oct 29 23:31:51.511079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.511050    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-openshift-ca\") pod \"04413b7d-e0ce-42b6-874c-fc5152590b95\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") "
Oct 29 23:31:51.511262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.511218    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04413b7d-e0ce-42b6-874c-fc5152590b95-kube-api-access-qnx4s\" (UniqueName: \"kubernetes.io/projected/04413b7d-e0ce-42b6-874c-fc5152590b95-kube-api-access-qnx4s\") pod \"04413b7d-e0ce-42b6-874c-fc5152590b95\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") "
Oct 29 23:31:51.511403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.511287    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-xks-ca\") pod \"04413b7d-e0ce-42b6-874c-fc5152590b95\" (UID: \"04413b7d-e0ce-42b6-874c-fc5152590b95\") "
Oct 29 23:31:51.511403 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.511374    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-openshift-ca" pod "04413b7d-e0ce-42b6-874c-fc5152590b95" (UID: "04413b7d-e0ce-42b6-874c-fc5152590b95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:51.511622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.511604    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:51.514800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.513376    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04413b7d-e0ce-42b6-874c-fc5152590b95-kube-api-access-qnx4s" pod "04413b7d-e0ce-42b6-874c-fc5152590b95" (UID: "04413b7d-e0ce-42b6-874c-fc5152590b95"). InnerVolumeSpecName "kube-api-access-qnx4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:51.514800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.513547    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-xks-ca" pod "04413b7d-e0ce-42b6-874c-fc5152590b95" (UID: "04413b7d-e0ce-42b6-874c-fc5152590b95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:51.613006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.612975    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04413b7d-e0ce-42b6-874c-fc5152590b95-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:51.613006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:51.613001    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnx4s\" (UniqueName: \"kubernetes.io/projected/04413b7d-e0ce-42b6-874c-fc5152590b95-kube-api-access-qnx4s\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:31:52.289493 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:52.289461    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8"
Oct 29 23:31:52.289671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:52.289476    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0a03df-b24f8" event={"ID":"04413b7d-e0ce-42b6-874c-fc5152590b95","Type":"ContainerDied","Data":"cfbfb0984cd1d2bae8c089bb380ee7ec259a073ecace8be28c4deaf02eedbeac"}
Oct 29 23:31:52.289671 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:52.289577    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfbfb0984cd1d2bae8c089bb380ee7ec259a073ecace8be28c4deaf02eedbeac"
Oct 29 23:31:55.512173 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:55.512142    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"]
Oct 29 23:31:55.528551 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:55.528526    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-59eefa-qz5d2"]
Oct 29 23:31:56.488025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:56.487989    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb265e4f-895b-41d0-8898-e40f24e1ff10" path="/var/lib/kubelet/pods/fb265e4f-895b-41d0-8898-e40f24e1ff10/volumes"
Oct 29 23:31:57.920867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:57.920828    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" podUID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:31:58.159949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.159914    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"]
Oct 29 23:31:58.160410 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160397    4827 cpu_manager.go:414] "containerMap: removing container" podUID="04413b7d-e0ce-42b6-874c-fc5152590b95" containerName="revoke-keys"
Oct 29 23:31:58.160461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160411    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04413b7d-e0ce-42b6-874c-fc5152590b95" containerName="revoke-keys"
Oct 29 23:31:58.160461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160436    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main"
Oct 29 23:31:58.160461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160441    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main"
Oct 29 23:31:58.160461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160450    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main"
Oct 29 23:31:58.160461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160456    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main"
Oct 29 23:31:58.160619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160468    4827 cpu_manager.go:414] "containerMap: removing container" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main"
Oct 29 23:31:58.160619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160473    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main"
Oct 29 23:31:58.160619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160553    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="04413b7d-e0ce-42b6-874c-fc5152590b95" containerName="revoke-keys"
Oct 29 23:31:58.160619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160564    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2081f6b2-15ff-4280-9d56-917613a6af5e" containerName="main"
Oct 29 23:31:58.160619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160569    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1833b55-765c-43e9-b608-a1c9ca16ca37" containerName="main"
Oct 29 23:31:58.160619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.160575    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5e2cbd-6e8f-47e0-a282-743b6525c169" containerName="main"
Oct 29 23:31:58.165625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.165606    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.179028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.179005    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:58.181744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.181716    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"]
Oct 29 23:31:58.189069 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.189040    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:31:58.292713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.292675    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlgs7\" (UniqueName: \"kubernetes.io/projected/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-kube-api-access-hlgs7\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.292917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.292733    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.293012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.292972    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.394126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.394090    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgs7\" (UniqueName: \"kubernetes.io/projected/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-kube-api-access-hlgs7\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.394308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.394158    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.394308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.394279    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.395003 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.394979    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.396614 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.396594    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.416419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.416392    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgs7\" (UniqueName: \"kubernetes.io/projected/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-kube-api-access-hlgs7\") pod \"maas-api-revoke-sub-6e461ae2-t89mx\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.478741 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.478681    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:31:58.777184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:58.777158    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"]
Oct 29 23:31:58.779091 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:31:58.779056    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71c9cc0_c97c_4e58_b7e3_c7ab502c337e.slice/crio-58b6bdaf60d60489cb7dfb90812e397356ac873f7b7c8fb12938cc9c33ee1b0f WatchSource:0}: Error finding container 58b6bdaf60d60489cb7dfb90812e397356ac873f7b7c8fb12938cc9c33ee1b0f: Status 404 returned error can't find the container with id 58b6bdaf60d60489cb7dfb90812e397356ac873f7b7c8fb12938cc9c33ee1b0f
Oct 29 23:31:59.334352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:59.334266    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx" event={"ID":"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e","Type":"ContainerStarted","Data":"9f09f49ee7a73fdd35beeb17ba5502df730b860c96b3458a50299dee2e2265b3"}
Oct 29 23:31:59.334352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:59.334343    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx" event={"ID":"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e","Type":"ContainerStarted","Data":"58b6bdaf60d60489cb7dfb90812e397356ac873f7b7c8fb12938cc9c33ee1b0f"}
Oct 29 23:31:59.371945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:31:59.371891    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx" podStartSLOduration=1.371877636 podStartE2EDuration="1.371877636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:59.369659733 +0000 UTC m=+2954.992570753" watchObservedRunningTime="2026-10-29 23:31:59.371877636 +0000 UTC m=+2954.994788702"
Oct 29 23:32:00.341422 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:00.341384    4827 generic.go:356] "Generic (PLEG): container finished" podID="d71c9cc0-c97c-4e58-b7e3-c7ab502c337e" containerID="9f09f49ee7a73fdd35beeb17ba5502df730b860c96b3458a50299dee2e2265b3" exitCode=0
Oct 29 23:32:00.341840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:00.341472    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx" event={"ID":"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e","Type":"ContainerDied","Data":"9f09f49ee7a73fdd35beeb17ba5502df730b860c96b3458a50299dee2e2265b3"}
Oct 29 23:32:00.871270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:00.871240    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"]
Oct 29 23:32:00.874807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:00.874757    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-575d59-9slxj"]
Oct 29 23:32:01.533385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.533356    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:32:01.644145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.644096    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"]
Oct 29 23:32:01.645722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.645695    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-openshift-ca\") pod \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") "
Oct 29 23:32:01.646359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.646338    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-xks-ca\") pod \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") "
Oct 29 23:32:01.646743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.646720    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-kube-api-access-hlgs7\" (UniqueName: \"kubernetes.io/projected/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-kube-api-access-hlgs7\") pod \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\" (UID: \"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e\") "
Oct 29 23:32:01.647161 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.646270    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-openshift-ca" pod "d71c9cc0-c97c-4e58-b7e3-c7ab502c337e" (UID: "d71c9cc0-c97c-4e58-b7e3-c7ab502c337e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:01.650934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.650912    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:01.661222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.659551    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-xks-ca" pod "d71c9cc0-c97c-4e58-b7e3-c7ab502c337e" (UID: "d71c9cc0-c97c-4e58-b7e3-c7ab502c337e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:01.661222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.660946    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-kube-api-access-hlgs7" pod "d71c9cc0-c97c-4e58-b7e3-c7ab502c337e" (UID: "d71c9cc0-c97c-4e58-b7e3-c7ab502c337e"). InnerVolumeSpecName "kube-api-access-hlgs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:01.768791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.754531    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlgs7\" (UniqueName: \"kubernetes.io/projected/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-kube-api-access-hlgs7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:01.768791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:01.754588    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c9cc0-c97c-4e58-b7e3-c7ab502c337e-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:02.357167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:02.357137    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx"
Oct 29 23:32:02.357364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:02.357177    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e461ae2-t89mx" event={"ID":"d71c9cc0-c97c-4e58-b7e3-c7ab502c337e","Type":"ContainerDied","Data":"58b6bdaf60d60489cb7dfb90812e397356ac873f7b7c8fb12938cc9c33ee1b0f"}
Oct 29 23:32:02.357364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:02.357274    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b6bdaf60d60489cb7dfb90812e397356ac873f7b7c8fb12938cc9c33ee1b0f"
Oct 29 23:32:02.488590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:02.488552    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a282c7fc-3a9c-4314-840c-a1edbf26f12f" path="/var/lib/kubelet/pods/a282c7fc-3a9c-4314-840c-a1edbf26f12f/volumes"
Oct 29 23:32:02.823195 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:02.823159    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:32:03.537476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:03.537444    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"]
Oct 29 23:32:03.537760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:03.537708    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy" containerID="cri-o://9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236" gracePeriod=30
Oct 29 23:32:04.375507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.375476    4827 generic.go:356] "Generic (PLEG): container finished" podID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" containerID="cc8189589deaba35c57b7cd4a97b7927c789cacec26685fae8ad10c8488a30e4" exitCode=0
Oct 29 23:32:04.376022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.375547    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" event={"ID":"feefe299-2d10-4139-a3b1-29a6dc9a68c2","Type":"ContainerDied","Data":"cc8189589deaba35c57b7cd4a97b7927c789cacec26685fae8ad10c8488a30e4"}
Oct 29 23:32:04.447251 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.447181    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 23:32:04.614745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.611186    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feefe299-2d10-4139-a3b1-29a6dc9a68c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feefe299-2d10-4139-a3b1-29a6dc9a68c2-plugins-config-volume\") pod \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") "
Oct 29 23:32:04.614745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.613626    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feefe299-2d10-4139-a3b1-29a6dc9a68c2-plugins-config-volume" pod "feefe299-2d10-4139-a3b1-29a6dc9a68c2" (UID: "feefe299-2d10-4139-a3b1-29a6dc9a68c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:04.614745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.613834    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feefe299-2d10-4139-a3b1-29a6dc9a68c2-kube-api-access-mthgh\" (UniqueName: \"kubernetes.io/projected/feefe299-2d10-4139-a3b1-29a6dc9a68c2-kube-api-access-mthgh\") pod \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\" (UID: \"feefe299-2d10-4139-a3b1-29a6dc9a68c2\") "
Oct 29 23:32:04.614745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.614296    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feefe299-2d10-4139-a3b1-29a6dc9a68c2-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:04.617326 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.617284    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feefe299-2d10-4139-a3b1-29a6dc9a68c2-kube-api-access-mthgh" pod "feefe299-2d10-4139-a3b1-29a6dc9a68c2" (UID: "feefe299-2d10-4139-a3b1-29a6dc9a68c2"). InnerVolumeSpecName "kube-api-access-mthgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:04.716046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:04.715974    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mthgh\" (UniqueName: \"kubernetes.io/projected/feefe299-2d10-4139-a3b1-29a6dc9a68c2-kube-api-access-mthgh\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:05.383681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:05.383648    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"
Oct 29 23:32:05.384233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:05.383646    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl" event={"ID":"feefe299-2d10-4139-a3b1-29a6dc9a68c2","Type":"ContainerDied","Data":"3d2aa064c105887129e1a6e3f7de103637edf154f390064e153ab069b4aab884"}
Oct 29 23:32:05.384233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:05.383865    4827 scope.go:122] "RemoveContainer" containerID="cc8189589deaba35c57b7cd4a97b7927c789cacec26685fae8ad10c8488a30e4"
Oct 29 23:32:05.433959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:05.433924    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"]
Oct 29 23:32:05.444419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:05.444391    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-575d59-94d668dc9-dgwpl"]
Oct 29 23:32:06.487294 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:06.487262    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" path="/var/lib/kubelet/pods/feefe299-2d10-4139-a3b1-29a6dc9a68c2/volumes"
Oct 29 23:32:08.265819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.265754    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"]
Oct 29 23:32:08.266331 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.266023    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy" containerID="cri-o://2cd1ee35af340535ea9763552f3699770021f91562983b9c958cc622c2fe5cfe" gracePeriod=30
Oct 29 23:32:08.789979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.789954    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:32:08.889744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.889720    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-envoy\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.889913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.889777    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-podinfo\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.889913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.889824    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-credential-socket\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.889913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.889877    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-kube-api-access-sjqmk\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-kube-api-access-sjqmk\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.890122 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890099    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-data\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.890222 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890186    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-certs\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.890282 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890208    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-credential-socket" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:08.890340 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890289    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-token\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-token\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.890400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890341    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-socket\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.890454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890389    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-data" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:08.890454 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890411    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace4e306-cefd-462b-9276-812fa8fb1c7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace4e306-cefd-462b-9276-812fa8fb1c7b-istiod-ca-cert\") pod \"ace4e306-cefd-462b-9276-812fa8fb1c7b\" (UID: \"ace4e306-cefd-462b-9276-812fa8fb1c7b\") "
Oct 29 23:32:08.890555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890511    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-certs" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:08.890818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.890709    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-socket" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:08.891046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.891010    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace4e306-cefd-462b-9276-812fa8fb1c7b-istiod-ca-cert" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:08.891156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.891045    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.891156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.891067    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.891156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.891084    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.891156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.891097    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.892552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.892524    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-podinfo" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:32:08.892679 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.892633    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-kube-api-access-sjqmk" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "kube-api-access-sjqmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:08.892851 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.892834    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-envoy" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:08.893099 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.893072    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-token" pod "ace4e306-cefd-462b-9276-812fa8fb1c7b" (UID: "ace4e306-cefd-462b-9276-812fa8fb1c7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:08.992499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.992466    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.992499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.992496    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace4e306-cefd-462b-9276-812fa8fb1c7b-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.992499 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.992508    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.992823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.992519    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace4e306-cefd-462b-9276-812fa8fb1c7b-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.992823 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:08.992530    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqmk\" (UniqueName: \"kubernetes.io/projected/ace4e306-cefd-462b-9276-812fa8fb1c7b-kube-api-access-sjqmk\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:09.446114 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.446079    4827 generic.go:356] "Generic (PLEG): container finished" podID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerID="9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236" exitCode=0
Oct 29 23:32:09.446576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.446126    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" event={"ID":"ace4e306-cefd-462b-9276-812fa8fb1c7b","Type":"ContainerDied","Data":"9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236"}
Oct 29 23:32:09.446576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.446132    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"
Oct 29 23:32:09.446576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.446169    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn" event={"ID":"ace4e306-cefd-462b-9276-812fa8fb1c7b","Type":"ContainerDied","Data":"04069da5864ff44e713e32bf2055fa5af573319134e60624822f12814f4eb965"}
Oct 29 23:32:09.446576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.446193    4827 scope.go:122] "RemoveContainer" containerID="9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236"
Oct 29 23:32:09.457104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.457082    4827 scope.go:122] "RemoveContainer" containerID="85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4"
Oct 29 23:32:09.466611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.466592    4827 scope.go:122] "RemoveContainer" containerID="9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236"
Oct 29 23:32:09.466914 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:09.466896    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236\": container with ID starting with 9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236 not found: ID does not exist" containerID="9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236"
Oct 29 23:32:09.466976 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.466922    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236"} err="failed to get container status \"9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236\": rpc error: code = NotFound desc = could not find container \"9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236\": container with ID starting with 9d5c52859efb1cc83029100cb12ba0af2a79b8579b82ad35d4dc012c56773236 not found: ID does not exist"
Oct 29 23:32:09.466976 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.466937    4827 scope.go:122] "RemoveContainer" containerID="85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4"
Oct 29 23:32:09.467162 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:09.467144    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4\": container with ID starting with 85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4 not found: ID does not exist" containerID="85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4"
Oct 29 23:32:09.467206 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.467167    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4"} err="failed to get container status \"85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4\": rpc error: code = NotFound desc = could not find container \"85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4\": container with ID starting with 85418d3a75af63fb553474496d4dc6487012365366507e687f49e6dc84319dc4 not found: ID does not exist"
Oct 29 23:32:09.494349 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.494327    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"]
Oct 29 23:32:09.504867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:09.504845    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-59eefa-openshift-default-57f6c5889f-m98nn"]
Oct 29 23:32:10.487815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:10.487758    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" path="/var/lib/kubelet/pods/ace4e306-cefd-462b-9276-812fa8fb1c7b/volumes"
Oct 29 23:32:13.530793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.530544    4827 generic.go:356] "Generic (PLEG): container finished" podID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerID="2cd1ee35af340535ea9763552f3699770021f91562983b9c958cc622c2fe5cfe" exitCode=0
Oct 29 23:32:13.530793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.530695    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" event={"ID":"e99488f4-d71f-4173-a8e7-7def27f6298c","Type":"ContainerDied","Data":"2cd1ee35af340535ea9763552f3699770021f91562983b9c958cc622c2fe5cfe"}
Oct 29 23:32:13.628224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.628192    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766000    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-credential-socket\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766060    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-socket\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766090    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-token\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-token\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766174    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-kube-api-access-zgl9z\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-kube-api-access-zgl9z\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766198    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-certs\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766310    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-data\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766339    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-podinfo\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766367    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-envoy\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.770821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.766478    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99488f4-d71f-4173-a8e7-7def27f6298c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99488f4-d71f-4173-a8e7-7def27f6298c-istiod-ca-cert\") pod \"e99488f4-d71f-4173-a8e7-7def27f6298c\" (UID: \"e99488f4-d71f-4173-a8e7-7def27f6298c\") "
Oct 29 23:32:13.775382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.775115    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99488f4-d71f-4173-a8e7-7def27f6298c-istiod-ca-cert" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:13.784784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.776099    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-certs" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:13.784784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.776292    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-socket" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:13.784784 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.776464    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-credential-socket" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:13.791364 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.791305    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-kube-api-access-zgl9z" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "kube-api-access-zgl9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:13.792234 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.792205    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-envoy" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:13.793184 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.793134    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-data" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:13.794786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.794720    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-podinfo" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:32:13.796490 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.796442    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-token" pod "e99488f4-d71f-4173-a8e7-7def27f6298c" (UID: "e99488f4-d71f-4173-a8e7-7def27f6298c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:13.868711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868627    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.868711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868665    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.868711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868680    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.868711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868694    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99488f4-d71f-4173-a8e7-7def27f6298c-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.868711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868707    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.869110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868721    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.869110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868737    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.869110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868752    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgl9z\" (UniqueName: \"kubernetes.io/projected/e99488f4-d71f-4173-a8e7-7def27f6298c-kube-api-access-zgl9z\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:13.869110 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:13.868802    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99488f4-d71f-4173-a8e7-7def27f6298c-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:14.538274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:14.538239    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl" event={"ID":"e99488f4-d71f-4173-a8e7-7def27f6298c","Type":"ContainerDied","Data":"7750cb302c1c08a78abfbdcde6236c0bb74e6c0b8e329ebe6e67829e24a7d8b9"}
Oct 29 23:32:14.538274 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:14.538259    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"
Oct 29 23:32:14.538715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:14.538315    4827 scope.go:122] "RemoveContainer" containerID="2cd1ee35af340535ea9763552f3699770021f91562983b9c958cc622c2fe5cfe"
Oct 29 23:32:14.612335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:14.612299    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"]
Oct 29 23:32:14.622929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:14.622903    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ql2jl"]
Oct 29 23:32:14.628953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:14.628930    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"]
Oct 29 23:32:16.487842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:16.487810    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" path="/var/lib/kubelet/pods/e99488f4-d71f-4173-a8e7-7def27f6298c/volumes"
Oct 29 23:32:16.685697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:16.685661    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf" podUID="a38f74ea-d450-44f5-8c7c-7a110cbc34c7" containerName="main" containerID="cri-o://8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162" gracePeriod=45
Oct 29 23:32:16.832083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:16.832047    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf" podUID="a38f74ea-d450-44f5-8c7c-7a110cbc34c7" containerName="main" probeResult="failure" output="Get \"https://10.134.1.57:8000/ready\": dial tcp 10.134.1.57:8000: connect: connection refused"
Oct 29 23:32:16.946060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:16.946035    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:32:17.023647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.023615    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-model-cache\") pod \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") "
Oct 29 23:32:17.023912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.023684    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tls-certs\") pod \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") "
Oct 29 23:32:17.023912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.023736    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tmp-dir\") pod \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") "
Oct 29 23:32:17.023912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.023788    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-dshm\") pod \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") "
Oct 29 23:32:17.023912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.023827    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-home\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-home\") pod \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") "
Oct 29 23:32:17.023912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.023875    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-kube-api-access-kzpxg\" (UniqueName: \"kubernetes.io/projected/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-kube-api-access-kzpxg\") pod \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\" (UID: \"a38f74ea-d450-44f5-8c7c-7a110cbc34c7\") "
Oct 29 23:32:17.024238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.023954    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-model-cache" pod "a38f74ea-d450-44f5-8c7c-7a110cbc34c7" (UID: "a38f74ea-d450-44f5-8c7c-7a110cbc34c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:17.024238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.024001    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tmp-dir" pod "a38f74ea-d450-44f5-8c7c-7a110cbc34c7" (UID: "a38f74ea-d450-44f5-8c7c-7a110cbc34c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:17.024238 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.024177    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-home" pod "a38f74ea-d450-44f5-8c7c-7a110cbc34c7" (UID: "a38f74ea-d450-44f5-8c7c-7a110cbc34c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:17.024532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.024513    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:17.024532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.024534    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:17.024682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.024542    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:17.026430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.026401    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tls-certs" pod "a38f74ea-d450-44f5-8c7c-7a110cbc34c7" (UID: "a38f74ea-d450-44f5-8c7c-7a110cbc34c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:17.026532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.026492    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-dshm" pod "a38f74ea-d450-44f5-8c7c-7a110cbc34c7" (UID: "a38f74ea-d450-44f5-8c7c-7a110cbc34c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:17.026532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.026496    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-kube-api-access-kzpxg" pod "a38f74ea-d450-44f5-8c7c-7a110cbc34c7" (UID: "a38f74ea-d450-44f5-8c7c-7a110cbc34c7"). InnerVolumeSpecName "kube-api-access-kzpxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:17.125708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.125679    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:17.125708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.125705    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:17.125928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.125717    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzpxg\" (UniqueName: \"kubernetes.io/projected/a38f74ea-d450-44f5-8c7c-7a110cbc34c7-kube-api-access-kzpxg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:17.558551 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.558469    4827 generic.go:356] "Generic (PLEG): container finished" podID="a38f74ea-d450-44f5-8c7c-7a110cbc34c7" containerID="8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162" exitCode=0
Oct 29 23:32:17.558551 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.558511    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"
Oct 29 23:32:17.559029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.558552    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf" event={"ID":"a38f74ea-d450-44f5-8c7c-7a110cbc34c7","Type":"ContainerDied","Data":"8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162"}
Oct 29 23:32:17.559029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.558600    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf" event={"ID":"a38f74ea-d450-44f5-8c7c-7a110cbc34c7","Type":"ContainerDied","Data":"91f83e6b5d2cd3eb23955f81a4bfd0d55c98203e3e3b0656be1f89feae82dcf6"}
Oct 29 23:32:17.559029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.558618    4827 scope.go:122] "RemoveContainer" containerID="8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162"
Oct 29 23:32:17.569307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.569285    4827 scope.go:122] "RemoveContainer" containerID="8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162"
Oct 29 23:32:17.569561 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:17.569541    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162\": container with ID starting with 8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162 not found: ID does not exist" containerID="8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162"
Oct 29 23:32:17.569640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.569572    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162"} err="failed to get container status \"8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162\": rpc error: code = NotFound desc = could not find container \"8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162\": container with ID starting with 8601fd74ccf6c61bbf8416d21f2895800edc8be2b9c2500099b7b5325b45f162 not found: ID does not exist"
Oct 29 23:32:17.605648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.605619    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"]
Oct 29 23:32:17.616090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:17.616059    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/sub-test-model-af237d-kserve-79d79c8995-bsfmf"]
Oct 29 23:32:18.488094 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:18.488064    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38f74ea-d450-44f5-8c7c-7a110cbc34c7" path="/var/lib/kubelet/pods/a38f74ea-d450-44f5-8c7c-7a110cbc34c7/volumes"
Oct 29 23:32:19.541918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.541879    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"]
Oct 29 23:32:19.542554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542532    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a38f74ea-d450-44f5-8c7c-7a110cbc34c7" containerName="main"
Oct 29 23:32:19.542655 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542556    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38f74ea-d450-44f5-8c7c-7a110cbc34c7" containerName="main"
Oct 29 23:32:19.542655 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542590    4827 cpu_manager.go:414] "containerMap: removing container" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy"
Oct 29 23:32:19.542655 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542642    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542663    4827 cpu_manager.go:414] "containerMap: removing container" podUID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" containerName="payload-pre-processing"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542672    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" containerName="payload-pre-processing"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542682    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542690    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542708    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542716    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542736    4827 cpu_manager.go:414] "containerMap: removing container" podUID="d71c9cc0-c97c-4e58-b7e3-c7ab502c337e" containerName="revoke-keys"
Oct 29 23:32:19.542860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542744    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71c9cc0-c97c-4e58-b7e3-c7ab502c337e" containerName="revoke-keys"
Oct 29 23:32:19.543286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542907    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99488f4-d71f-4173-a8e7-7def27f6298c" containerName="istio-proxy"
Oct 29 23:32:19.543286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542921    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="feefe299-2d10-4139-a3b1-29a6dc9a68c2" containerName="payload-pre-processing"
Oct 29 23:32:19.543286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542933    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy"
Oct 29 23:32:19.543286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542945    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace4e306-cefd-462b-9276-812fa8fb1c7b" containerName="istio-proxy"
Oct 29 23:32:19.543286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542957    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71c9cc0-c97c-4e58-b7e3-c7ab502c337e" containerName="revoke-keys"
Oct 29 23:32:19.543286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.542970    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38f74ea-d450-44f5-8c7c-7a110cbc34c7" containerName="main"
Oct 29 23:32:19.564341 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.564309    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.591408 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.591365    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"]
Oct 29 23:32:19.618940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.618912    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:19.619065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.618975    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:32:19.630108 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.630065    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" podUID="5b0efb42-deeb-4e75-b829-c9fd12d39024" containerName="payload-pre-processing" containerID="cri-o://8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10" gracePeriod=25
Oct 29 23:32:19.655642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.655619    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.655760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.655659    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.655760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.655698    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxz89\" (UniqueName: \"kubernetes.io/projected/475634d9-c3d3-4b50-b57e-7f3462c19aea-kube-api-access-sxz89\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.756870 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.756841    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.757047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.756880    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.757047 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.756910    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxz89\" (UniqueName: \"kubernetes.io/projected/475634d9-c3d3-4b50-b57e-7f3462c19aea-kube-api-access-sxz89\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.757527 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.757498    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.759286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.759266    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.778701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.778675    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxz89\" (UniqueName: \"kubernetes.io/projected/475634d9-c3d3-4b50-b57e-7f3462c19aea-kube-api-access-sxz89\") pod \"maas-api-revoke-sub-afe349f3-b5dvl\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:19.917139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:19.917115    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:20.221228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:20.221202    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"]
Oct 29 23:32:20.223194 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:32:20.223164    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475634d9_c3d3_4b50_b57e_7f3462c19aea.slice/crio-05fbe8dede2ca8569d4796e50ffd140404fcfbf5134d329baae9e96cf3a7b13b WatchSource:0}: Error finding container 05fbe8dede2ca8569d4796e50ffd140404fcfbf5134d329baae9e96cf3a7b13b: Status 404 returned error can't find the container with id 05fbe8dede2ca8569d4796e50ffd140404fcfbf5134d329baae9e96cf3a7b13b
Oct 29 23:32:20.612727 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:20.612696    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl" event={"ID":"475634d9-c3d3-4b50-b57e-7f3462c19aea","Type":"ContainerStarted","Data":"370a5d55ae28d08d5578d0705859382e93128462cb930121522241b51d924d96"}
Oct 29 23:32:20.613166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:20.612749    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl" event={"ID":"475634d9-c3d3-4b50-b57e-7f3462c19aea","Type":"ContainerStarted","Data":"05fbe8dede2ca8569d4796e50ffd140404fcfbf5134d329baae9e96cf3a7b13b"}
Oct 29 23:32:20.649436 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:20.649378    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl" podStartSLOduration=1.6493655189999998 podStartE2EDuration="1.649365519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:20.64856754 +0000 UTC m=+2976.271478560" watchObservedRunningTime="2026-10-29 23:32:20.649365519 +0000 UTC m=+2976.272276539"
Oct 29 23:32:21.620289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:21.620257    4827 generic.go:356] "Generic (PLEG): container finished" podID="475634d9-c3d3-4b50-b57e-7f3462c19aea" containerID="370a5d55ae28d08d5578d0705859382e93128462cb930121522241b51d924d96" exitCode=0
Oct 29 23:32:21.620697 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:21.620335    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl" event={"ID":"475634d9-c3d3-4b50-b57e-7f3462c19aea","Type":"ContainerDied","Data":"370a5d55ae28d08d5578d0705859382e93128462cb930121522241b51d924d96"}
Oct 29 23:32:22.775753 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.775722    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:22.897136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.897054    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-xks-ca\") pod \"475634d9-c3d3-4b50-b57e-7f3462c19aea\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") "
Oct 29 23:32:22.897309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.897163    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475634d9-c3d3-4b50-b57e-7f3462c19aea-kube-api-access-sxz89\" (UniqueName: \"kubernetes.io/projected/475634d9-c3d3-4b50-b57e-7f3462c19aea-kube-api-access-sxz89\") pod \"475634d9-c3d3-4b50-b57e-7f3462c19aea\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") "
Oct 29 23:32:22.897309 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.897240    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-openshift-ca\") pod \"475634d9-c3d3-4b50-b57e-7f3462c19aea\" (UID: \"475634d9-c3d3-4b50-b57e-7f3462c19aea\") "
Oct 29 23:32:22.897682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.897653    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-openshift-ca" pod "475634d9-c3d3-4b50-b57e-7f3462c19aea" (UID: "475634d9-c3d3-4b50-b57e-7f3462c19aea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:22.899153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.899125    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-xks-ca" pod "475634d9-c3d3-4b50-b57e-7f3462c19aea" (UID: "475634d9-c3d3-4b50-b57e-7f3462c19aea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:22.899290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.899205    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475634d9-c3d3-4b50-b57e-7f3462c19aea-kube-api-access-sxz89" pod "475634d9-c3d3-4b50-b57e-7f3462c19aea" (UID: "475634d9-c3d3-4b50-b57e-7f3462c19aea"). InnerVolumeSpecName "kube-api-access-sxz89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:22.999245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.999215    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:22.999245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.999244    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475634d9-c3d3-4b50-b57e-7f3462c19aea-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:22.999245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:22.999253    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxz89\" (UniqueName: \"kubernetes.io/projected/475634d9-c3d3-4b50-b57e-7f3462c19aea-kube-api-access-sxz89\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:23.633332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:23.633302    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl" event={"ID":"475634d9-c3d3-4b50-b57e-7f3462c19aea","Type":"ContainerDied","Data":"05fbe8dede2ca8569d4796e50ffd140404fcfbf5134d329baae9e96cf3a7b13b"}
Oct 29 23:32:23.633512 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:23.633349    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05fbe8dede2ca8569d4796e50ffd140404fcfbf5134d329baae9e96cf3a7b13b"
Oct 29 23:32:23.633512 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:23.633310    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe349f3-b5dvl"
Oct 29 23:32:27.752659 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:27.752620    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" podUID="5b0efb42-deeb-4e75-b829-c9fd12d39024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:28.593208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:28.593175    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"]
Oct 29 23:32:28.626834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:28.626803    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"]
Oct 29 23:32:30.944368 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:30.944333    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"]
Oct 29 23:32:30.944786 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:30.944565    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp" podUID="b9534671-b20f-432d-a550-bcf1d18bc280" containerName="maas-api" containerID="cri-o://1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71" gracePeriod=30
Oct 29 23:32:31.243721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.243339    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:32:31.309419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.309372    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9534671-b20f-432d-a550-bcf1d18bc280-kube-api-access-7vlqm\" (UniqueName: \"kubernetes.io/projected/b9534671-b20f-432d-a550-bcf1d18bc280-kube-api-access-7vlqm\") pod \"b9534671-b20f-432d-a550-bcf1d18bc280\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") "
Oct 29 23:32:31.309687 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.309466    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-maas-api-tls\") pod \"b9534671-b20f-432d-a550-bcf1d18bc280\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") "
Oct 29 23:32:31.309687 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.309545    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9534671-b20f-432d-a550-bcf1d18bc280-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9534671-b20f-432d-a550-bcf1d18bc280-openshift-service-ca\") pod \"b9534671-b20f-432d-a550-bcf1d18bc280\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") "
Oct 29 23:32:31.309687 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.309651    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-metrics-certs\") pod \"b9534671-b20f-432d-a550-bcf1d18bc280\" (UID: \"b9534671-b20f-432d-a550-bcf1d18bc280\") "
Oct 29 23:32:31.310523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.310468    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9534671-b20f-432d-a550-bcf1d18bc280-openshift-service-ca" pod "b9534671-b20f-432d-a550-bcf1d18bc280" (UID: "b9534671-b20f-432d-a550-bcf1d18bc280"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:31.348806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.332277    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9534671-b20f-432d-a550-bcf1d18bc280-kube-api-access-7vlqm" pod "b9534671-b20f-432d-a550-bcf1d18bc280" (UID: "b9534671-b20f-432d-a550-bcf1d18bc280"). InnerVolumeSpecName "kube-api-access-7vlqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:31.348806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.334446    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-metrics-certs" pod "b9534671-b20f-432d-a550-bcf1d18bc280" (UID: "b9534671-b20f-432d-a550-bcf1d18bc280"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:31.357982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.357939    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-maas-api-tls" pod "b9534671-b20f-432d-a550-bcf1d18bc280" (UID: "b9534671-b20f-432d-a550-bcf1d18bc280"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:31.411131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.411094    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlqm\" (UniqueName: \"kubernetes.io/projected/b9534671-b20f-432d-a550-bcf1d18bc280-kube-api-access-7vlqm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:31.411131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.411123    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:31.411131 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.411134    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9534671-b20f-432d-a550-bcf1d18bc280-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:31.411535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.411143    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9534671-b20f-432d-a550-bcf1d18bc280-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:31.686428 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.686397    4827 generic.go:356] "Generic (PLEG): container finished" podID="b9534671-b20f-432d-a550-bcf1d18bc280" containerID="1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71" exitCode=0
Oct 29 23:32:31.686658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.686456    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"
Oct 29 23:32:31.686658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.686488    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp" event={"ID":"b9534671-b20f-432d-a550-bcf1d18bc280","Type":"ContainerDied","Data":"1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71"}
Oct 29 23:32:31.686658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.686559    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp" event={"ID":"b9534671-b20f-432d-a550-bcf1d18bc280","Type":"ContainerDied","Data":"4ffe81fbbd66954a24eba10e3e311e9ee8fff47dc54ef7053a1e68819a0de3bf"}
Oct 29 23:32:31.686658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.686589    4827 scope.go:122] "RemoveContainer" containerID="1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71"
Oct 29 23:32:31.698840 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.698822    4827 scope.go:122] "RemoveContainer" containerID="1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71"
Oct 29 23:32:31.699165 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:31.699143    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71\": container with ID starting with 1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71 not found: ID does not exist" containerID="1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71"
Oct 29 23:32:31.699253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.699171    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71"} err="failed to get container status \"1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71\": rpc error: code = NotFound desc = could not find container \"1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71\": container with ID starting with 1ec67c8533298d56b39ad4f31a3bef7516b4c080bf980099e8d6ae72aadfca71 not found: ID does not exist"
Oct 29 23:32:31.733793 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.733749    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"]
Oct 29 23:32:31.744369 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:31.744319    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-af237d-687689865f-gkfrp"]
Oct 29 23:32:32.492852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:32.492794    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9534671-b20f-432d-a550-bcf1d18bc280" path="/var/lib/kubelet/pods/b9534671-b20f-432d-a550-bcf1d18bc280/volumes"
Oct 29 23:32:32.822225 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:32.822135    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:32:33.593881 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:33.593821    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" podUID="440801e3-e740-4278-bcdb-0bd4418b9d2e" containerName="payload-processing" containerID="cri-o://864a0e6c3d7988b857bd39e7986b4d272f035b806ccfc41f2284229134286628" gracePeriod=25
Oct 29 23:32:33.627604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:33.627530    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" podUID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" containerName="payload-pre-processing" containerID="cri-o://54b37c43f28e6318d3cf97d2651e6b7c402e08a81a101445187d6e2a0d2fb892" gracePeriod=25
Oct 29 23:32:34.919073 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.919036    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"]
Oct 29 23:32:34.919816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.919782    4827 cpu_manager.go:414] "containerMap: removing container" podUID="b9534671-b20f-432d-a550-bcf1d18bc280" containerName="maas-api"
Oct 29 23:32:34.919816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.919803    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9534671-b20f-432d-a550-bcf1d18bc280" containerName="maas-api"
Oct 29 23:32:34.919950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.919828    4827 cpu_manager.go:414] "containerMap: removing container" podUID="475634d9-c3d3-4b50-b57e-7f3462c19aea" containerName="revoke-keys"
Oct 29 23:32:34.919950 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.919836    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475634d9-c3d3-4b50-b57e-7f3462c19aea" containerName="revoke-keys"
Oct 29 23:32:34.920025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.919964    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9534671-b20f-432d-a550-bcf1d18bc280" containerName="maas-api"
Oct 29 23:32:34.920025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.919977    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="475634d9-c3d3-4b50-b57e-7f3462c19aea" containerName="revoke-keys"
Oct 29 23:32:34.925071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.925050    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:34.940252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.940200    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"]
Oct 29 23:32:34.962253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.962223    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttv66\" (UniqueName: \"kubernetes.io/projected/458cf807-b753-42c7-9519-44c292b0af5b-kube-api-access-ttv66\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:34.962367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.962277    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:34.962367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.962336    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:34.985258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.985228    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:34.985531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:34.985513    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:32:35.063721 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.063687    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttv66\" (UniqueName: \"kubernetes.io/projected/458cf807-b753-42c7-9519-44c292b0af5b-kube-api-access-ttv66\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:35.063949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.063797    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:35.063949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.063908    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:35.064556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.064531    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:35.066317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.066285    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:35.085576 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.085547    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttv66\" (UniqueName: \"kubernetes.io/projected/458cf807-b753-42c7-9519-44c292b0af5b-kube-api-access-ttv66\") pod \"maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:35.238818 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.238720    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:35.540421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.540393    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"]
Oct 29 23:32:35.542191 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:32:35.542166    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458cf807_b753_42c7_9519_44c292b0af5b.slice/crio-b34c11c78836c1ab7748c6d27fcce69b9c4406117f507af27775b1710a53e94c WatchSource:0}: Error finding container b34c11c78836c1ab7748c6d27fcce69b9c4406117f507af27775b1710a53e94c: Status 404 returned error can't find the container with id b34c11c78836c1ab7748c6d27fcce69b9c4406117f507af27775b1710a53e94c
Oct 29 23:32:35.716334 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:35.716299    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds" event={"ID":"458cf807-b753-42c7-9519-44c292b0af5b","Type":"ContainerStarted","Data":"b34c11c78836c1ab7748c6d27fcce69b9c4406117f507af27775b1710a53e94c"}
Oct 29 23:32:36.477862 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:36.477836    4827 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458cf807_b753_42c7_9519_44c292b0af5b.slice/crio-85bc5aed938c93d8ddb40988f5ac79df91825ec082c724a819d81604a1d859c0.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:32:36.724313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:36.724226    4827 generic.go:356] "Generic (PLEG): container finished" podID="458cf807-b753-42c7-9519-44c292b0af5b" containerID="85bc5aed938c93d8ddb40988f5ac79df91825ec082c724a819d81604a1d859c0" exitCode=0
Oct 29 23:32:36.724472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:36.724327    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds" event={"ID":"458cf807-b753-42c7-9519-44c292b0af5b","Type":"ContainerDied","Data":"85bc5aed938c93d8ddb40988f5ac79df91825ec082c724a819d81604a1d859c0"}
Oct 29 23:32:37.753156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.753118    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" podUID="5b0efb42-deeb-4e75-b829-c9fd12d39024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:37.872252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.872229    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:37.917469 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.917403    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-openshift-ca\") pod \"458cf807-b753-42c7-9519-44c292b0af5b\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") "
Oct 29 23:32:37.917608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.917473    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-xks-ca\") pod \"458cf807-b753-42c7-9519-44c292b0af5b\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") "
Oct 29 23:32:37.917608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.917514    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458cf807-b753-42c7-9519-44c292b0af5b-kube-api-access-ttv66\" (UniqueName: \"kubernetes.io/projected/458cf807-b753-42c7-9519-44c292b0af5b-kube-api-access-ttv66\") pod \"458cf807-b753-42c7-9519-44c292b0af5b\" (UID: \"458cf807-b753-42c7-9519-44c292b0af5b\") "
Oct 29 23:32:37.917839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.917814    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-openshift-ca" pod "458cf807-b753-42c7-9519-44c292b0af5b" (UID: "458cf807-b753-42c7-9519-44c292b0af5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:37.920532 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.920492    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-xks-ca" pod "458cf807-b753-42c7-9519-44c292b0af5b" (UID: "458cf807-b753-42c7-9519-44c292b0af5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:37.920914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:37.920888    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458cf807-b753-42c7-9519-44c292b0af5b-kube-api-access-ttv66" pod "458cf807-b753-42c7-9519-44c292b0af5b" (UID: "458cf807-b753-42c7-9519-44c292b0af5b"). InnerVolumeSpecName "kube-api-access-ttv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:38.019290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:38.019258    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttv66\" (UniqueName: \"kubernetes.io/projected/458cf807-b753-42c7-9519-44c292b0af5b-kube-api-access-ttv66\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:38.019290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:38.019286    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:38.019290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:38.019300    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458cf807-b753-42c7-9519-44c292b0af5b-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:38.738491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:38.738465    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"
Oct 29 23:32:38.738667 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:38.738462    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds" event={"ID":"458cf807-b753-42c7-9519-44c292b0af5b","Type":"ContainerDied","Data":"b34c11c78836c1ab7748c6d27fcce69b9c4406117f507af27775b1710a53e94c"}
Oct 29 23:32:38.738667 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:38.738602    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b34c11c78836c1ab7748c6d27fcce69b9c4406117f507af27775b1710a53e94c"
Oct 29 23:32:39.710285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:39.710252    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"]
Oct 29 23:32:39.727703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:39.727671    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"]
Oct 29 23:32:39.831148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:39.831116    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"]
Oct 29 23:32:39.831357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:39.831329    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" containerID="cri-o://c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff" gracePeriod=30
Oct 29 23:32:39.976246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:39.976204    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:32:40.041913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.041882    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0efb42-deeb-4e75-b829-c9fd12d39024-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0efb42-deeb-4e75-b829-c9fd12d39024-plugins-config-volume\") pod \"5b0efb42-deeb-4e75-b829-c9fd12d39024\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") "
Oct 29 23:32:40.041913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.041918    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0efb42-deeb-4e75-b829-c9fd12d39024-kube-api-access-x7j4x\" (UniqueName: \"kubernetes.io/projected/5b0efb42-deeb-4e75-b829-c9fd12d39024-kube-api-access-x7j4x\") pod \"5b0efb42-deeb-4e75-b829-c9fd12d39024\" (UID: \"5b0efb42-deeb-4e75-b829-c9fd12d39024\") "
Oct 29 23:32:40.042354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.042320    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0efb42-deeb-4e75-b829-c9fd12d39024-plugins-config-volume" pod "5b0efb42-deeb-4e75-b829-c9fd12d39024" (UID: "5b0efb42-deeb-4e75-b829-c9fd12d39024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:40.044186 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.044159    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0efb42-deeb-4e75-b829-c9fd12d39024-kube-api-access-x7j4x" pod "5b0efb42-deeb-4e75-b829-c9fd12d39024" (UID: "5b0efb42-deeb-4e75-b829-c9fd12d39024"). InnerVolumeSpecName "kube-api-access-x7j4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:40.143957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.143925    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0efb42-deeb-4e75-b829-c9fd12d39024-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:40.143957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.143953    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7j4x\" (UniqueName: \"kubernetes.io/projected/5b0efb42-deeb-4e75-b829-c9fd12d39024-kube-api-access-x7j4x\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:40.407498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.407464    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"]
Oct 29 23:32:40.407986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.407969    4827 cpu_manager.go:414] "containerMap: removing container" podUID="458cf807-b753-42c7-9519-44c292b0af5b" containerName="revoke-keys"
Oct 29 23:32:40.407986 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.407984    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458cf807-b753-42c7-9519-44c292b0af5b" containerName="revoke-keys"
Oct 29 23:32:40.408166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.408024    4827 cpu_manager.go:414] "containerMap: removing container" podUID="5b0efb42-deeb-4e75-b829-c9fd12d39024" containerName="payload-pre-processing"
Oct 29 23:32:40.408166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.408030    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0efb42-deeb-4e75-b829-c9fd12d39024" containerName="payload-pre-processing"
Oct 29 23:32:40.408166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.408105    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0efb42-deeb-4e75-b829-c9fd12d39024" containerName="payload-pre-processing"
Oct 29 23:32:40.408166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.408116    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="458cf807-b753-42c7-9519-44c292b0af5b" containerName="revoke-keys"
Oct 29 23:32:40.411471 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.411450    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.431416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.431384    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"]
Oct 29 23:32:40.453836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.448622    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbk2q\" (UniqueName: \"kubernetes.io/projected/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-kube-api-access-lbk2q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.453836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.448688    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.453836 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.448837    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.501259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.501227    4827 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:32:40.501447 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.501287    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:32:40.517451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.517421    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:40.517601 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.517567    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:32:40.549926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.549898    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.550097 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.550016    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.550201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.550185    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbk2q\" (UniqueName: \"kubernetes.io/projected/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-kube-api-access-lbk2q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.550685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.550663    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.552328 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.552299    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.572070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.572049    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbk2q\" (UniqueName: \"kubernetes.io/projected/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-kube-api-access-lbk2q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.724980 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.724886    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:40.754856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.754815    4827 generic.go:356] "Generic (PLEG): container finished" podID="5b0efb42-deeb-4e75-b829-c9fd12d39024" containerID="8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10" exitCode=0
Oct 29 23:32:40.755028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.754855    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" event={"ID":"5b0efb42-deeb-4e75-b829-c9fd12d39024","Type":"ContainerDied","Data":"8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10"}
Oct 29 23:32:40.755028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.754833    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"
Oct 29 23:32:40.755028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.754912    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd" event={"ID":"5b0efb42-deeb-4e75-b829-c9fd12d39024","Type":"ContainerDied","Data":"44575a7690ab6fa78ed979d81d82516d35d4232303fc00583714c28c880e19d3"}
Oct 29 23:32:40.755028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.754942    4827 scope.go:122] "RemoveContainer" containerID="8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10"
Oct 29 23:32:40.766372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.766325    4827 scope.go:122] "RemoveContainer" containerID="8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10"
Oct 29 23:32:40.766676 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:40.766650    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10\": container with ID starting with 8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10 not found: ID does not exist" containerID="8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10"
Oct 29 23:32:40.766797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.766678    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10"} err="failed to get container status \"8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10\": rpc error: code = NotFound desc = could not find container \"8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10\": container with ID starting with 8d48a4445aad9408a94f669af01f307dc1344a0c1cfbf30ee6a66e39fa4cbb10 not found: ID does not exist"
Oct 29 23:32:40.803026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.802995    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"]
Oct 29 23:32:40.807917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:40.807890    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e170ed-5bbcc8d78c-bp9gd"]
Oct 29 23:32:41.028982 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:41.028728    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"]
Oct 29 23:32:41.031558 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:32:41.031533    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253a98e8_2f2e_4fc9_9527_31ea0f70e4f5.slice/crio-28abdeb993a1eba84e7cd36c72dc461a74b2c8542c89da0fb2a79e87549a8cc6 WatchSource:0}: Error finding container 28abdeb993a1eba84e7cd36c72dc461a74b2c8542c89da0fb2a79e87549a8cc6: Status 404 returned error can't find the container with id 28abdeb993a1eba84e7cd36c72dc461a74b2c8542c89da0fb2a79e87549a8cc6
Oct 29 23:32:41.763852 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:41.763813    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn" event={"ID":"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5","Type":"ContainerStarted","Data":"2f00d367d9e4d3ce792ff3b5a056d741e092da924e35cacf462b0a3ec4b92ad5"}
Oct 29 23:32:41.764290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:41.763879    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn" event={"ID":"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5","Type":"ContainerStarted","Data":"28abdeb993a1eba84e7cd36c72dc461a74b2c8542c89da0fb2a79e87549a8cc6"}
Oct 29 23:32:41.803513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:41.803452    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn" podStartSLOduration=1.803437298 podStartE2EDuration="1.803437298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:41.800519474 +0000 UTC m=+2997.423430496" watchObservedRunningTime="2026-10-29 23:32:41.803437298 +0000 UTC m=+2997.426348308"
Oct 29 23:32:42.491791 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:42.490698    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0efb42-deeb-4e75-b829-c9fd12d39024" path="/var/lib/kubelet/pods/5b0efb42-deeb-4e75-b829-c9fd12d39024/volumes"
Oct 29 23:32:42.652531 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:42.652459    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" podUID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:42.652748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:42.652539    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" podUID="440801e3-e740-4278-bcdb-0bd4418b9d2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:42.771259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:42.771174    4827 generic.go:356] "Generic (PLEG): container finished" podID="253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" containerID="2f00d367d9e4d3ce792ff3b5a056d741e092da924e35cacf462b0a3ec4b92ad5" exitCode=0
Oct 29 23:32:42.771259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:42.771210    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn" event={"ID":"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5","Type":"ContainerDied","Data":"2f00d367d9e4d3ce792ff3b5a056d741e092da924e35cacf462b0a3ec4b92ad5"}
Oct 29 23:32:43.933060 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:43.933034    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:44.022837 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.022809    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-kube-api-access-lbk2q\" (UniqueName: \"kubernetes.io/projected/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-kube-api-access-lbk2q\") pod \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") "
Oct 29 23:32:44.022990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.022917    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-openshift-ca\") pod \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") "
Oct 29 23:32:44.022990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.022983    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-xks-ca\") pod \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\" (UID: \"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5\") "
Oct 29 23:32:44.023246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.023220    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-openshift-ca" pod "253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" (UID: "253a98e8-2f2e-4fc9-9527-31ea0f70e4f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:44.024978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.024956    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-xks-ca" pod "253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" (UID: "253a98e8-2f2e-4fc9-9527-31ea0f70e4f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:44.024978 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.024971    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-kube-api-access-lbk2q" pod "253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" (UID: "253a98e8-2f2e-4fc9-9527-31ea0f70e4f5"). InnerVolumeSpecName "kube-api-access-lbk2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:44.124458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.124431    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:44.124458 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.124457    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:44.124632 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.124469    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbk2q\" (UniqueName: \"kubernetes.io/projected/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5-kube-api-access-lbk2q\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:44.784811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.784726    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"
Oct 29 23:32:44.784811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.784746    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn" event={"ID":"253a98e8-2f2e-4fc9-9527-31ea0f70e4f5","Type":"ContainerDied","Data":"28abdeb993a1eba84e7cd36c72dc461a74b2c8542c89da0fb2a79e87549a8cc6"}
Oct 29 23:32:44.784811 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:44.784807    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28abdeb993a1eba84e7cd36c72dc461a74b2c8542c89da0fb2a79e87549a8cc6"
Oct 29 23:32:45.097667 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.097645    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:32:45.235266 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235229    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-podinfo\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235283    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-token\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-token\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235340    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-credential-socket\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235459 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235386    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-certs\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235515    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-envoy\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235553    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istiod-ca-cert\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235601    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-data\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235663 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235634    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-socket\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235704    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-kube-api-access-fh662\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-kube-api-access-fh662\") pod \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\" (UID: \"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8\") "
Oct 29 23:32:45.235921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235708    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-certs" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:45.235921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235717    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-credential-socket" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:45.236078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235887    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-data" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:45.236078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235954    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-socket" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:45.236078 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.235974    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istiod-ca-cert" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:45.236270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.236248    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.236323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.236276    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.236323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.236289    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.236323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.236303    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.236426 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.236331    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.237907 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.237882    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-podinfo" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:32:45.238035 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.237998    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-kube-api-access-fh662" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "kube-api-access-fh662". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:45.238104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.238068    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-token" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:45.238104 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.238067    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-envoy" pod "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" (UID: "cec51f58-8af3-4cdb-ba1c-9ca1edd497a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:45.337240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.337164    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh662\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-kube-api-access-fh662\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.337240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.337189    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.337240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.337198    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.337240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.337207    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:45.384734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.384702    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"]
Oct 29 23:32:45.423630 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.423585    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"]
Oct 29 23:32:45.792295 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.792265    4827 generic.go:356] "Generic (PLEG): container finished" podID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerID="c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff" exitCode=0
Oct 29 23:32:45.792461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.792316    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"
Oct 29 23:32:45.792461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.792318    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" event={"ID":"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8","Type":"ContainerDied","Data":"c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff"}
Oct 29 23:32:45.792461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.792426    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw" event={"ID":"cec51f58-8af3-4cdb-ba1c-9ca1edd497a8","Type":"ContainerDied","Data":"99005227fb870e94b19d4270567b9ed73056b74d5848294064c1e4bbf0d16d5e"}
Oct 29 23:32:45.792461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.792448    4827 scope.go:122] "RemoveContainer" containerID="c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff"
Oct 29 23:32:45.802685 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.802668    4827 scope.go:122] "RemoveContainer" containerID="c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff"
Oct 29 23:32:45.803020 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:45.803001    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff\": container with ID starting with c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff not found: ID does not exist" containerID="c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff"
Oct 29 23:32:45.803096 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.803025    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff"} err="failed to get container status \"c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff\": rpc error: code = NotFound desc = could not find container \"c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff\": container with ID starting with c7fa710596497b3f9147eebd601479179b0fb7f33b7e95491732a5bc9c8ee7ff not found: ID does not exist"
Oct 29 23:32:45.840714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.840689    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"]
Oct 29 23:32:45.844323 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:45.844301    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-8v4mw"]
Oct 29 23:32:46.488046 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:46.488007    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" path="/var/lib/kubelet/pods/cec51f58-8af3-4cdb-ba1c-9ca1edd497a8/volumes"
Oct 29 23:32:46.606568 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:46.606537    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"]
Oct 29 23:32:46.643452 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:46.643416    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"]
Oct 29 23:32:47.366815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:47.366759    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:32:47.505841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:47.498937    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sbt6_dc8ed141-dd58-45a2-841e-22c1d8eb19c1/ovn-acl-logging/0.log"
Oct 29 23:32:50.394517 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:50.387836    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerName="payload-processing" containerID="cri-o://4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0" gracePeriod=25
Oct 29 23:32:50.424525 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:50.424465    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerName="payload-pre-processing" containerID="cri-o://d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813" gracePeriod=25
Oct 29 23:32:50.431990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:50.431960    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:50.432118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:50.431960    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:51.547743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.547626    4827 scope.go:122] "RemoveContainer" containerID="7fd1159b1012cd1340b6b149245bb6c2f25aee0e531c1dac732db2e02e8f7bf0"
Oct 29 23:32:51.607682 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.607645    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" podUID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" containerName="payload-processing" containerID="cri-o://b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf" gracePeriod=25
Oct 29 23:32:51.644307 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.644269    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" podUID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" containerName="payload-pre-processing" containerID="cri-o://d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780" gracePeriod=25
Oct 29 23:32:51.718126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.718094    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"]
Oct 29 23:32:51.718589 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.718576    4827 cpu_manager.go:414] "containerMap: removing container" podUID="253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" containerName="revoke-keys"
Oct 29 23:32:51.718638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.718591    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" containerName="revoke-keys"
Oct 29 23:32:51.718638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.718606    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy"
Oct 29 23:32:51.718638 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.718611    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy"
Oct 29 23:32:51.718743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.718679    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec51f58-8af3-4cdb-ba1c-9ca1edd497a8" containerName="istio-proxy"
Oct 29 23:32:51.718743 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.718692    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" containerName="revoke-keys"
Oct 29 23:32:51.723877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.723856    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.737619 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.737592    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:51.740129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.740110    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:32:51.741736 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.741713    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.741878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.741754    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.741878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.741796    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xk72\" (UniqueName: \"kubernetes.io/projected/2e87c67a-7b47-4aeb-8818-55d48813a7bc-kube-api-access-9xk72\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.753853 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.753829    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"]
Oct 29 23:32:51.843555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.843474    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.843555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.843513    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.843555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.843533    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xk72\" (UniqueName: \"kubernetes.io/projected/2e87c67a-7b47-4aeb-8818-55d48813a7bc-kube-api-access-9xk72\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.844136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.844113    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.845921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.845897    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:51.865949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:51.865925    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xk72\" (UniqueName: \"kubernetes.io/projected/2e87c67a-7b47-4aeb-8818-55d48813a7bc-kube-api-access-9xk72\") pod \"maas-api-revoke-sub-a0ed4fa4-8xfvr\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:52.038979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.038935    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:52.362683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.362644    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"]
Oct 29 23:32:52.364055 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:32:52.364025    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e87c67a_7b47_4aeb_8818_55d48813a7bc.slice/crio-7c72128be483da37a9972e19b5db06bc84b6c0bfdb66f114e720abf3eb4e836b WatchSource:0}: Error finding container 7c72128be483da37a9972e19b5db06bc84b6c0bfdb66f114e720abf3eb4e836b: Status 404 returned error can't find the container with id 7c72128be483da37a9972e19b5db06bc84b6c0bfdb66f114e720abf3eb4e836b
Oct 29 23:32:52.652058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.652008    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" podUID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:52.652593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.652274    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" podUID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:52.652593 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.652570    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" podUID="440801e3-e740-4278-bcdb-0bd4418b9d2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:52.652725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.652635    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" podUID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:52.842416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.842382    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr" event={"ID":"2e87c67a-7b47-4aeb-8818-55d48813a7bc","Type":"ContainerStarted","Data":"9b5151050e0675a6d4f8076bfbfe92b08ba062e0e9c74549f8bbb25f0ac6d1be"}
Oct 29 23:32:52.842416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.842432    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr" event={"ID":"2e87c67a-7b47-4aeb-8818-55d48813a7bc","Type":"ContainerStarted","Data":"7c72128be483da37a9972e19b5db06bc84b6c0bfdb66f114e720abf3eb4e836b"}
Oct 29 23:32:52.881050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:52.881003    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr" podStartSLOduration=1.8809892480000001 podStartE2EDuration="1.880989248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:52.878826444 +0000 UTC m=+3008.501737461" watchObservedRunningTime="2026-10-29 23:32:52.880989248 +0000 UTC m=+3008.503900268"
Oct 29 23:32:53.407198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.407167    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"]
Oct 29 23:32:53.407450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.407395    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf" podUID="c64dd95b-d357-488e-9aa4-8d3376e2cba3" containerName="maas-api" containerID="cri-o://c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb" gracePeriod=30
Oct 29 23:32:53.744416 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.744392    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:32:53.770859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.764466    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-metrics-certs\") pod \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") "
Oct 29 23:32:53.770859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.764553    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64dd95b-d357-488e-9aa4-8d3376e2cba3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c64dd95b-d357-488e-9aa4-8d3376e2cba3-openshift-service-ca\") pod \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") "
Oct 29 23:32:53.770859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.764595    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64dd95b-d357-488e-9aa4-8d3376e2cba3-kube-api-access-fmqb5\" (UniqueName: \"kubernetes.io/projected/c64dd95b-d357-488e-9aa4-8d3376e2cba3-kube-api-access-fmqb5\") pod \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") "
Oct 29 23:32:53.770859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.764651    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-maas-api-tls\") pod \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\" (UID: \"c64dd95b-d357-488e-9aa4-8d3376e2cba3\") "
Oct 29 23:32:53.770859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.765982    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64dd95b-d357-488e-9aa4-8d3376e2cba3-openshift-service-ca" pod "c64dd95b-d357-488e-9aa4-8d3376e2cba3" (UID: "c64dd95b-d357-488e-9aa4-8d3376e2cba3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:53.776732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.775876    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-metrics-certs" pod "c64dd95b-d357-488e-9aa4-8d3376e2cba3" (UID: "c64dd95b-d357-488e-9aa4-8d3376e2cba3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:53.776732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.775920    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64dd95b-d357-488e-9aa4-8d3376e2cba3-kube-api-access-fmqb5" pod "c64dd95b-d357-488e-9aa4-8d3376e2cba3" (UID: "c64dd95b-d357-488e-9aa4-8d3376e2cba3"). InnerVolumeSpecName "kube-api-access-fmqb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:53.789970 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.785884    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-maas-api-tls" pod "c64dd95b-d357-488e-9aa4-8d3376e2cba3" (UID: "c64dd95b-d357-488e-9aa4-8d3376e2cba3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:53.884815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.873011    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:53.884815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.873048    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c64dd95b-d357-488e-9aa4-8d3376e2cba3-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:53.884815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.873065    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmqb5\" (UniqueName: \"kubernetes.io/projected/c64dd95b-d357-488e-9aa4-8d3376e2cba3-kube-api-access-fmqb5\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:53.884815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.873080    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c64dd95b-d357-488e-9aa4-8d3376e2cba3-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:53.900012 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.899977    4827 generic.go:356] "Generic (PLEG): container finished" podID="c64dd95b-d357-488e-9aa4-8d3376e2cba3" containerID="c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb" exitCode=0
Oct 29 23:32:53.900265 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.900146    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"
Oct 29 23:32:53.900608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.900558    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf" event={"ID":"c64dd95b-d357-488e-9aa4-8d3376e2cba3","Type":"ContainerDied","Data":"c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb"}
Oct 29 23:32:53.900702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.900641    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf" event={"ID":"c64dd95b-d357-488e-9aa4-8d3376e2cba3","Type":"ContainerDied","Data":"222c95bd39d43ac0cd5fc6106076c4d2e9762a9f8c4c5e522a2e8c04cba0a726"}
Oct 29 23:32:53.900702 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.900673    4827 scope.go:122] "RemoveContainer" containerID="c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb"
Oct 29 23:32:53.915579 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.910575    4827 generic.go:356] "Generic (PLEG): container finished" podID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" containerID="54b37c43f28e6318d3cf97d2651e6b7c402e08a81a101445187d6e2a0d2fb892" exitCode=0
Oct 29 23:32:53.915579 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.910727    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" event={"ID":"eca901e7-95bf-4acf-89ad-4d06c2e616fa","Type":"ContainerDied","Data":"54b37c43f28e6318d3cf97d2651e6b7c402e08a81a101445187d6e2a0d2fb892"}
Oct 29 23:32:53.920552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.920518    4827 generic.go:356] "Generic (PLEG): container finished" podID="440801e3-e740-4278-bcdb-0bd4418b9d2e" containerID="864a0e6c3d7988b857bd39e7986b4d272f035b806ccfc41f2284229134286628" exitCode=0
Oct 29 23:32:53.920718 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.920697    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" event={"ID":"440801e3-e740-4278-bcdb-0bd4418b9d2e","Type":"ContainerDied","Data":"864a0e6c3d7988b857bd39e7986b4d272f035b806ccfc41f2284229134286628"}
Oct 29 23:32:53.923119 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.923090    4827 generic.go:356] "Generic (PLEG): container finished" podID="2e87c67a-7b47-4aeb-8818-55d48813a7bc" containerID="9b5151050e0675a6d4f8076bfbfe92b08ba062e0e9c74549f8bbb25f0ac6d1be" exitCode=0
Oct 29 23:32:53.923243 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.923192    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr" event={"ID":"2e87c67a-7b47-4aeb-8818-55d48813a7bc","Type":"ContainerDied","Data":"9b5151050e0675a6d4f8076bfbfe92b08ba062e0e9c74549f8bbb25f0ac6d1be"}
Oct 29 23:32:53.927153 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.927137    4827 scope.go:122] "RemoveContainer" containerID="c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb"
Oct 29 23:32:53.927491 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:32:53.927464    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb\": container with ID starting with c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb not found: ID does not exist" containerID="c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb"
Oct 29 23:32:53.927575 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.927500    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb"} err="failed to get container status \"c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb\": rpc error: code = NotFound desc = could not find container \"c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb\": container with ID starting with c6810dd36739940ebb3a7c541d24a8e509207ac71afb764900940bbdbfa3b9cb not found: ID does not exist"
Oct 29 23:32:53.967835 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.964478    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"]
Oct 29 23:32:53.969224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.968901    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4504f1-c8556555d-n2kxf"]
Oct 29 23:32:53.977919 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:53.977886    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:32:54.028279 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.028215    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:32:54.175929 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.175897    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca901e7-95bf-4acf-89ad-4d06c2e616fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca901e7-95bf-4acf-89ad-4d06c2e616fa-plugins-config-volume\") pod \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") "
Oct 29 23:32:54.176121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.175947    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440801e3-e740-4278-bcdb-0bd4418b9d2e-kube-api-access-xfvh6\" (UniqueName: \"kubernetes.io/projected/440801e3-e740-4278-bcdb-0bd4418b9d2e-kube-api-access-xfvh6\") pod \"440801e3-e740-4278-bcdb-0bd4418b9d2e\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") "
Oct 29 23:32:54.176121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.175980    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca901e7-95bf-4acf-89ad-4d06c2e616fa-kube-api-access-qrqr4\" (UniqueName: \"kubernetes.io/projected/eca901e7-95bf-4acf-89ad-4d06c2e616fa-kube-api-access-qrqr4\") pod \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\" (UID: \"eca901e7-95bf-4acf-89ad-4d06c2e616fa\") "
Oct 29 23:32:54.176121 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.176067    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440801e3-e740-4278-bcdb-0bd4418b9d2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440801e3-e740-4278-bcdb-0bd4418b9d2e-plugins-config-volume\") pod \"440801e3-e740-4278-bcdb-0bd4418b9d2e\" (UID: \"440801e3-e740-4278-bcdb-0bd4418b9d2e\") "
Oct 29 23:32:54.176363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.176337    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca901e7-95bf-4acf-89ad-4d06c2e616fa-plugins-config-volume" pod "eca901e7-95bf-4acf-89ad-4d06c2e616fa" (UID: "eca901e7-95bf-4acf-89ad-4d06c2e616fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:54.176694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.176668    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440801e3-e740-4278-bcdb-0bd4418b9d2e-plugins-config-volume" pod "440801e3-e740-4278-bcdb-0bd4418b9d2e" (UID: "440801e3-e740-4278-bcdb-0bd4418b9d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:54.176694 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.176674    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca901e7-95bf-4acf-89ad-4d06c2e616fa-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:54.178646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.178618    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca901e7-95bf-4acf-89ad-4d06c2e616fa-kube-api-access-qrqr4" pod "eca901e7-95bf-4acf-89ad-4d06c2e616fa" (UID: "eca901e7-95bf-4acf-89ad-4d06c2e616fa"). InnerVolumeSpecName "kube-api-access-qrqr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:54.181070 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.181048    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440801e3-e740-4278-bcdb-0bd4418b9d2e-kube-api-access-xfvh6" pod "440801e3-e740-4278-bcdb-0bd4418b9d2e" (UID: "440801e3-e740-4278-bcdb-0bd4418b9d2e"). InnerVolumeSpecName "kube-api-access-xfvh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:54.289879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.289473    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqr4\" (UniqueName: \"kubernetes.io/projected/eca901e7-95bf-4acf-89ad-4d06c2e616fa-kube-api-access-qrqr4\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:54.289879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.289515    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440801e3-e740-4278-bcdb-0bd4418b9d2e-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:54.289879 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.289528    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfvh6\" (UniqueName: \"kubernetes.io/projected/440801e3-e740-4278-bcdb-0bd4418b9d2e-kube-api-access-xfvh6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:54.488953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.488918    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64dd95b-d357-488e-9aa4-8d3376e2cba3" path="/var/lib/kubelet/pods/c64dd95b-d357-488e-9aa4-8d3376e2cba3/volumes"
Oct 29 23:32:54.730952 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.730903    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main" containerID="cri-o://e98325f42e095b01b51307278226bde121d67d89cb9cfb1142f365cda962c27b" gracePeriod=45
Oct 29 23:32:54.750513 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.750481    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main" containerID="cri-o://f2b56a6b983165a972ac8baeb3ace6d9121d54d8eb54bb6f9ad207cbd2c95dfb" gracePeriod=45
Oct 29 23:32:54.939395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.939365    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"
Oct 29 23:32:54.940023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.939368    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6" event={"ID":"440801e3-e740-4278-bcdb-0bd4418b9d2e","Type":"ContainerDied","Data":"6bef23f3ed121be78539ced5183c0dc37277fe2453379e61a88e76b82406f7a6"}
Oct 29 23:32:54.940170 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.940060    4827 scope.go:122] "RemoveContainer" containerID="864a0e6c3d7988b857bd39e7986b4d272f035b806ccfc41f2284229134286628"
Oct 29 23:32:54.943599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.943448    4827 generic.go:356] "Generic (PLEG): container finished" podID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerID="f2b56a6b983165a972ac8baeb3ace6d9121d54d8eb54bb6f9ad207cbd2c95dfb" exitCode=0
Oct 29 23:32:54.943599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.943516    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" event={"ID":"705240b9-b2bf-4a99-b78e-6216e4e0350e","Type":"ContainerDied","Data":"f2b56a6b983165a972ac8baeb3ace6d9121d54d8eb54bb6f9ad207cbd2c95dfb"}
Oct 29 23:32:54.947825 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.947792    4827 generic.go:356] "Generic (PLEG): container finished" podID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerID="e98325f42e095b01b51307278226bde121d67d89cb9cfb1142f365cda962c27b" exitCode=0
Oct 29 23:32:54.947931 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.947876    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" event={"ID":"ea51e23a-12d9-46d6-9445-5b48cc12459d","Type":"ContainerDied","Data":"e98325f42e095b01b51307278226bde121d67d89cb9cfb1142f365cda962c27b"}
Oct 29 23:32:54.949626 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.949605    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"
Oct 29 23:32:54.955198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.954220    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn" event={"ID":"eca901e7-95bf-4acf-89ad-4d06c2e616fa","Type":"ContainerDied","Data":"8f4deada4db3d4621285ee4241f6688c9736c196275a25b90f5a36dab7dcc76b"}
Oct 29 23:32:54.955198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:54.954837    4827 scope.go:122] "RemoveContainer" containerID="54b37c43f28e6318d3cf97d2651e6b7c402e08a81a101445187d6e2a0d2fb892"
Oct 29 23:32:55.009119 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.009031    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"]
Oct 29 23:32:55.015006 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.014960    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-af237d-59c468ff59-7tqwn"]
Oct 29 23:32:55.067726 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.065919    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"]
Oct 29 23:32:55.097363 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.097337    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af237d-76898b7798-kr4c6"]
Oct 29 23:32:55.114995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.114970    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:32:55.138873 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.138848    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:32:55.210318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.210274    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-home\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-home\") pod \"705240b9-b2bf-4a99-b78e-6216e4e0350e\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") "
Oct 29 23:32:55.210569 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.210550    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-home" pod "705240b9-b2bf-4a99-b78e-6216e4e0350e" (UID: "705240b9-b2bf-4a99-b78e-6216e4e0350e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.210725 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.210708    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea51e23a-12d9-46d6-9445-5b48cc12459d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea51e23a-12d9-46d6-9445-5b48cc12459d-tls-certs\") pod \"ea51e23a-12d9-46d6-9445-5b48cc12459d\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") "
Oct 29 23:32:55.211257 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211234    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea51e23a-12d9-46d6-9445-5b48cc12459d-kube-api-access-q5jbj\" (UniqueName: \"kubernetes.io/projected/ea51e23a-12d9-46d6-9445-5b48cc12459d-kube-api-access-q5jbj\") pod \"ea51e23a-12d9-46d6-9445-5b48cc12459d\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") "
Oct 29 23:32:55.211387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211282    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-home\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-home\") pod \"ea51e23a-12d9-46d6-9445-5b48cc12459d\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") "
Oct 29 23:32:55.211387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211332    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-tmp-dir\") pod \"705240b9-b2bf-4a99-b78e-6216e4e0350e\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") "
Oct 29 23:32:55.211387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211375    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-dshm\") pod \"ea51e23a-12d9-46d6-9445-5b48cc12459d\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") "
Oct 29 23:32:55.211574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211409    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705240b9-b2bf-4a99-b78e-6216e4e0350e-tls-certs\" (UniqueName: \"kubernetes.io/secret/705240b9-b2bf-4a99-b78e-6216e4e0350e-tls-certs\") pod \"705240b9-b2bf-4a99-b78e-6216e4e0350e\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") "
Oct 29 23:32:55.211574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211461    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-model-cache\") pod \"705240b9-b2bf-4a99-b78e-6216e4e0350e\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") "
Oct 29 23:32:55.211574 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211503    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-dshm\") pod \"705240b9-b2bf-4a99-b78e-6216e4e0350e\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") "
Oct 29 23:32:55.212167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211652    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705240b9-b2bf-4a99-b78e-6216e4e0350e-kube-api-access-8trd9\" (UniqueName: \"kubernetes.io/projected/705240b9-b2bf-4a99-b78e-6216e4e0350e-kube-api-access-8trd9\") pod \"705240b9-b2bf-4a99-b78e-6216e4e0350e\" (UID: \"705240b9-b2bf-4a99-b78e-6216e4e0350e\") "
Oct 29 23:32:55.212167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211735    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-model-cache\") pod \"ea51e23a-12d9-46d6-9445-5b48cc12459d\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") "
Oct 29 23:32:55.212167 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.211788    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-tmp-dir\") pod \"ea51e23a-12d9-46d6-9445-5b48cc12459d\" (UID: \"ea51e23a-12d9-46d6-9445-5b48cc12459d\") "
Oct 29 23:32:55.215956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.215938    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.216211 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.216197    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-tmp-dir" pod "ea51e23a-12d9-46d6-9445-5b48cc12459d" (UID: "ea51e23a-12d9-46d6-9445-5b48cc12459d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.217050 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.217023    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-model-cache" pod "705240b9-b2bf-4a99-b78e-6216e4e0350e" (UID: "705240b9-b2bf-4a99-b78e-6216e4e0350e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.218618 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.218421    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-model-cache" pod "ea51e23a-12d9-46d6-9445-5b48cc12459d" (UID: "ea51e23a-12d9-46d6-9445-5b48cc12459d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.219112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.218808    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-home" pod "ea51e23a-12d9-46d6-9445-5b48cc12459d" (UID: "ea51e23a-12d9-46d6-9445-5b48cc12459d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.220030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.219722    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-tmp-dir" pod "705240b9-b2bf-4a99-b78e-6216e4e0350e" (UID: "705240b9-b2bf-4a99-b78e-6216e4e0350e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.222113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.222090    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-dshm" pod "ea51e23a-12d9-46d6-9445-5b48cc12459d" (UID: "ea51e23a-12d9-46d6-9445-5b48cc12459d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.223159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.223123    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea51e23a-12d9-46d6-9445-5b48cc12459d-tls-certs" pod "ea51e23a-12d9-46d6-9445-5b48cc12459d" (UID: "ea51e23a-12d9-46d6-9445-5b48cc12459d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:55.223389 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.223352    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705240b9-b2bf-4a99-b78e-6216e4e0350e-tls-certs" pod "705240b9-b2bf-4a99-b78e-6216e4e0350e" (UID: "705240b9-b2bf-4a99-b78e-6216e4e0350e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:55.223498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.223420    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-dshm" pod "705240b9-b2bf-4a99-b78e-6216e4e0350e" (UID: "705240b9-b2bf-4a99-b78e-6216e4e0350e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:55.224126 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.224092    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea51e23a-12d9-46d6-9445-5b48cc12459d-kube-api-access-q5jbj" pod "ea51e23a-12d9-46d6-9445-5b48cc12459d" (UID: "ea51e23a-12d9-46d6-9445-5b48cc12459d"). InnerVolumeSpecName "kube-api-access-q5jbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:55.224562 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.224539    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705240b9-b2bf-4a99-b78e-6216e4e0350e-kube-api-access-8trd9" pod "705240b9-b2bf-4a99-b78e-6216e4e0350e" (UID: "705240b9-b2bf-4a99-b78e-6216e4e0350e"). InnerVolumeSpecName "kube-api-access-8trd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:55.229519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.229464    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:55.317259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317184    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-xks-ca\") pod \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") "
Oct 29 23:32:55.317259 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317233    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e87c67a-7b47-4aeb-8818-55d48813a7bc-kube-api-access-9xk72\" (UniqueName: \"kubernetes.io/projected/2e87c67a-7b47-4aeb-8818-55d48813a7bc-kube-api-access-9xk72\") pod \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") "
Oct 29 23:32:55.317448 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317265    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-openshift-ca\") pod \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\" (UID: \"2e87c67a-7b47-4aeb-8818-55d48813a7bc\") "
Oct 29 23:32:55.317661 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317632    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317667    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317677    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-openshift-ca" pod "2e87c67a-7b47-4aeb-8818-55d48813a7bc" (UID: "2e87c67a-7b47-4aeb-8818-55d48813a7bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:55.317723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317694    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trd9\" (UniqueName: \"kubernetes.io/projected/705240b9-b2bf-4a99-b78e-6216e4e0350e-kube-api-access-8trd9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317707    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317723 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317720    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317732    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea51e23a-12d9-46d6-9445-5b48cc12459d-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317743    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5jbj\" (UniqueName: \"kubernetes.io/projected/ea51e23a-12d9-46d6-9445-5b48cc12459d-kube-api-access-q5jbj\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317755    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317797    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705240b9-b2bf-4a99-b78e-6216e4e0350e-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317806    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea51e23a-12d9-46d6-9445-5b48cc12459d-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.317965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.317813    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705240b9-b2bf-4a99-b78e-6216e4e0350e-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.323151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.323110    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-xks-ca" pod "2e87c67a-7b47-4aeb-8818-55d48813a7bc" (UID: "2e87c67a-7b47-4aeb-8818-55d48813a7bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:55.323360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.323315    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e87c67a-7b47-4aeb-8818-55d48813a7bc-kube-api-access-9xk72" pod "2e87c67a-7b47-4aeb-8818-55d48813a7bc" (UID: "2e87c67a-7b47-4aeb-8818-55d48813a7bc"). InnerVolumeSpecName "kube-api-access-9xk72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:55.418902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.418866    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.418902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.418894    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e87c67a-7b47-4aeb-8818-55d48813a7bc-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.418902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.418904    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xk72\" (UniqueName: \"kubernetes.io/projected/2e87c67a-7b47-4aeb-8818-55d48813a7bc-kube-api-access-9xk72\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:55.759954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.759921    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"]
Oct 29 23:32:55.760418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.760196    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj" podUID="88837411-4699-4681-8ba8-7a8eada3293f" containerName="maas-api" containerID="cri-o://a69107d273f0731699cce9cf008e17f8ad2e0571eeede0f68350f0b131d51b56" gracePeriod=30
Oct 29 23:32:55.975690 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.975659    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr"
Oct 29 23:32:55.975984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.975686    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ed4fa4-8xfvr" event={"ID":"2e87c67a-7b47-4aeb-8818-55d48813a7bc","Type":"ContainerDied","Data":"7c72128be483da37a9972e19b5db06bc84b6c0bfdb66f114e720abf3eb4e836b"}
Oct 29 23:32:55.976155 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.976036    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c72128be483da37a9972e19b5db06bc84b6c0bfdb66f114e720abf3eb4e836b"
Oct 29 23:32:55.977816 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.977789    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8" event={"ID":"705240b9-b2bf-4a99-b78e-6216e4e0350e","Type":"ContainerDied","Data":"70351c9e813e27d149bcf21cb3bf6a1925c40165425888c6ddbf67e4a68e7e78"}
Oct 29 23:32:55.977953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.977813    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"
Oct 29 23:32:55.977953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.977836    4827 scope.go:122] "RemoveContainer" containerID="f2b56a6b983165a972ac8baeb3ace6d9121d54d8eb54bb6f9ad207cbd2c95dfb"
Oct 29 23:32:55.981016 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.980990    4827 generic.go:356] "Generic (PLEG): container finished" podID="88837411-4699-4681-8ba8-7a8eada3293f" containerID="a69107d273f0731699cce9cf008e17f8ad2e0571eeede0f68350f0b131d51b56" exitCode=0
Oct 29 23:32:55.981123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.981081    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj" event={"ID":"88837411-4699-4681-8ba8-7a8eada3293f","Type":"ContainerDied","Data":"a69107d273f0731699cce9cf008e17f8ad2e0571eeede0f68350f0b131d51b56"}
Oct 29 23:32:55.982610 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.982590    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm" event={"ID":"ea51e23a-12d9-46d6-9445-5b48cc12459d","Type":"ContainerDied","Data":"48764452d1abb2cea8dc3687d1e531a7f14ca36de058460232a5ccd84c65eba3"}
Oct 29 23:32:55.982707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.982657    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"
Oct 29 23:32:55.989434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:55.989416    4827 scope.go:122] "RemoveContainer" containerID="e98325f42e095b01b51307278226bde121d67d89cb9cfb1142f365cda962c27b"
Oct 29 23:32:56.054797 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.051459    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"]
Oct 29 23:32:56.069086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.069059    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/premium-w1-e131af-kserve-7dbccf6cd6-ndmm8"]
Oct 29 23:32:56.076949 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.076927    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:32:56.098644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.098617    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"]
Oct 29 23:32:56.102208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.102185    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-w1-e131af-kserve-85487c6d9-9vvhm"]
Oct 29 23:32:56.127856 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.127830    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-maas-api-tls\") pod \"88837411-4699-4681-8ba8-7a8eada3293f\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") "
Oct 29 23:32:56.127993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.127888    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-metrics-certs\") pod \"88837411-4699-4681-8ba8-7a8eada3293f\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") "
Oct 29 23:32:56.127993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.127960    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88837411-4699-4681-8ba8-7a8eada3293f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88837411-4699-4681-8ba8-7a8eada3293f-openshift-service-ca\") pod \"88837411-4699-4681-8ba8-7a8eada3293f\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") "
Oct 29 23:32:56.128137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.128047    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88837411-4699-4681-8ba8-7a8eada3293f-kube-api-access-crvnw\" (UniqueName: \"kubernetes.io/projected/88837411-4699-4681-8ba8-7a8eada3293f-kube-api-access-crvnw\") pod \"88837411-4699-4681-8ba8-7a8eada3293f\" (UID: \"88837411-4699-4681-8ba8-7a8eada3293f\") "
Oct 29 23:32:56.128397 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.128369    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88837411-4699-4681-8ba8-7a8eada3293f-openshift-service-ca" pod "88837411-4699-4681-8ba8-7a8eada3293f" (UID: "88837411-4699-4681-8ba8-7a8eada3293f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:56.130137 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.130115    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88837411-4699-4681-8ba8-7a8eada3293f-kube-api-access-crvnw" pod "88837411-4699-4681-8ba8-7a8eada3293f" (UID: "88837411-4699-4681-8ba8-7a8eada3293f"). InnerVolumeSpecName "kube-api-access-crvnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:56.130224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.130204    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-metrics-certs" pod "88837411-4699-4681-8ba8-7a8eada3293f" (UID: "88837411-4699-4681-8ba8-7a8eada3293f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:56.130272 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.130249    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-maas-api-tls" pod "88837411-4699-4681-8ba8-7a8eada3293f" (UID: "88837411-4699-4681-8ba8-7a8eada3293f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:56.229798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.229734    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvnw\" (UniqueName: \"kubernetes.io/projected/88837411-4699-4681-8ba8-7a8eada3293f-kube-api-access-crvnw\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:56.229798 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.229798    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-maas-api-tls\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:56.230007 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.229812    4827 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88837411-4699-4681-8ba8-7a8eada3293f-metrics-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:56.230007 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.229824    4827 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88837411-4699-4681-8ba8-7a8eada3293f-openshift-service-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:32:56.488329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.488287    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440801e3-e740-4278-bcdb-0bd4418b9d2e" path="/var/lib/kubelet/pods/440801e3-e740-4278-bcdb-0bd4418b9d2e/volumes"
Oct 29 23:32:56.488817 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.488797    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" path="/var/lib/kubelet/pods/705240b9-b2bf-4a99-b78e-6216e4e0350e/volumes"
Oct 29 23:32:56.489246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.489227    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" path="/var/lib/kubelet/pods/ea51e23a-12d9-46d6-9445-5b48cc12459d/volumes"
Oct 29 23:32:56.489866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.489843    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" path="/var/lib/kubelet/pods/eca901e7-95bf-4acf-89ad-4d06c2e616fa/volumes"
Oct 29 23:32:56.527681 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.527657    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"]
Oct 29 23:32:56.531001 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.530980    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-65e2d3-mj7xn"]
Oct 29 23:32:56.995145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.994941    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"
Oct 29 23:32:56.995145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.994956    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj" event={"ID":"88837411-4699-4681-8ba8-7a8eada3293f","Type":"ContainerDied","Data":"41c260c7575d7e9cceb0e7f3a2c2bc5ff6e545e56ffef2d16e9529c74a1211f2"}
Oct 29 23:32:56.995145 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:56.995026    4827 scope.go:122] "RemoveContainer" containerID="a69107d273f0731699cce9cf008e17f8ad2e0571eeede0f68350f0b131d51b56"
Oct 29 23:32:57.042390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:57.042356    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"]
Oct 29 23:32:57.052296 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:57.052271    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-65e2d3-558b5bfd66-2g2sj"]
Oct 29 23:32:58.487367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:58.487334    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253a98e8-2f2e-4fc9-9527-31ea0f70e4f5" path="/var/lib/kubelet/pods/253a98e8-2f2e-4fc9-9527-31ea0f70e4f5/volumes"
Oct 29 23:32:58.487917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:32:58.487798    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88837411-4699-4681-8ba8-7a8eada3293f" path="/var/lib/kubelet/pods/88837411-4699-4681-8ba8-7a8eada3293f/volumes"
Oct 29 23:33:00.432561 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:00.432524    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:00.433195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:00.432989    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:01.504805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:01.502005    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"]
Oct 29 23:33:01.504805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:01.502259    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy" containerID="cri-o://2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5" gracePeriod=30
Oct 29 23:33:02.083358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083324    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"]
Oct 29 23:33:02.083749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083734    4827 cpu_manager.go:414] "containerMap: removing container" podUID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" containerName="payload-pre-processing"
Oct 29 23:33:02.083827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083750    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" containerName="payload-pre-processing"
Oct 29 23:33:02.083827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083783    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2e87c67a-7b47-4aeb-8818-55d48813a7bc" containerName="revoke-keys"
Oct 29 23:33:02.083827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083790    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e87c67a-7b47-4aeb-8818-55d48813a7bc" containerName="revoke-keys"
Oct 29 23:33:02.083827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083815    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main"
Oct 29 23:33:02.083827 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083823    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083830    4827 cpu_manager.go:414] "containerMap: removing container" podUID="c64dd95b-d357-488e-9aa4-8d3376e2cba3" containerName="maas-api"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083835    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64dd95b-d357-488e-9aa4-8d3376e2cba3" containerName="maas-api"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083844    4827 cpu_manager.go:414] "containerMap: removing container" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083848    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083858    4827 cpu_manager.go:414] "containerMap: removing container" podUID="88837411-4699-4681-8ba8-7a8eada3293f" containerName="maas-api"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083863    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88837411-4699-4681-8ba8-7a8eada3293f" containerName="maas-api"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083871    4827 cpu_manager.go:414] "containerMap: removing container" podUID="440801e3-e740-4278-bcdb-0bd4418b9d2e" containerName="payload-processing"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083877    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440801e3-e740-4278-bcdb-0bd4418b9d2e" containerName="payload-processing"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083941    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="88837411-4699-4681-8ba8-7a8eada3293f" containerName="maas-api"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083951    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="440801e3-e740-4278-bcdb-0bd4418b9d2e" containerName="payload-processing"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083959    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca901e7-95bf-4acf-89ad-4d06c2e616fa" containerName="payload-pre-processing"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083966    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e87c67a-7b47-4aeb-8818-55d48813a7bc" containerName="revoke-keys"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083974    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea51e23a-12d9-46d6-9445-5b48cc12459d" containerName="main"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083979    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64dd95b-d357-488e-9aa4-8d3376e2cba3" containerName="maas-api"
Oct 29 23:33:02.083997 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.083984    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="705240b9-b2bf-4a99-b78e-6216e4e0350e" containerName="main"
Oct 29 23:33:02.088714 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.088695    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.106557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.106526    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"]
Oct 29 23:33:02.172854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.172820    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:33:02.173025 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.172878    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:33:02.188367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.188338    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.188518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.188372    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.188518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.188407    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dd59\" (UniqueName: \"kubernetes.io/projected/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-kube-api-access-4dd59\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.290053 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.290021    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.290246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.290061    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.290246 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.290095    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dd59\" (UniqueName: \"kubernetes.io/projected/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-kube-api-access-4dd59\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.291004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.290984    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.293590 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.293559    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.311882 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.311859    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dd59\" (UniqueName: \"kubernetes.io/projected/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-kube-api-access-4dd59\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.402792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.402754    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:02.671928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.669336    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" podUID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:02.671928 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.669831    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" podUID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:02.712842 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:02.712812    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"]
Oct 29 23:33:02.714797 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:02.714753    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb3fe42_9773_4c45_b2c6_a53c2f03ac10.slice/crio-96eafeffee03aad4e714ecf54d16e66045beb8a1bef5a541e8cdbdea5bf786b4 WatchSource:0}: Error finding container 96eafeffee03aad4e714ecf54d16e66045beb8a1bef5a541e8cdbdea5bf786b4: Status 404 returned error can't find the container with id 96eafeffee03aad4e714ecf54d16e66045beb8a1bef5a541e8cdbdea5bf786b4
Oct 29 23:33:02.823488 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:02.823452    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:33:03.032219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:03.032186    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp" event={"ID":"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10","Type":"ContainerStarted","Data":"96eafeffee03aad4e714ecf54d16e66045beb8a1bef5a541e8cdbdea5bf786b4"}
Oct 29 23:33:04.039432 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:04.039399    4827 generic.go:356] "Generic (PLEG): container finished" podID="3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" containerID="a75d12e1e5a0826cf54a5b0e6a3c92a05c43e14a165c0bb8593b0ba9857d1328" exitCode=0
Oct 29 23:33:04.039860 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:04.039444    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp" event={"ID":"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10","Type":"ContainerDied","Data":"a75d12e1e5a0826cf54a5b0e6a3c92a05c43e14a165c0bb8593b0ba9857d1328"}
Oct 29 23:33:05.184640 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.184613    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:05.223800 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.223739    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-kube-api-access-4dd59\" (UniqueName: \"kubernetes.io/projected/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-kube-api-access-4dd59\") pod \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") "
Oct 29 23:33:05.223955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.223867    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-xks-ca\") pod \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") "
Oct 29 23:33:05.223955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.223928    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-openshift-ca\") pod \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\" (UID: \"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10\") "
Oct 29 23:33:05.224539 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.224511    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-openshift-ca" pod "3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" (UID: "3fb3fe42-9773-4c45-b2c6-a53c2f03ac10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:05.226665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.226638    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-kube-api-access-4dd59" pod "3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" (UID: "3fb3fe42-9773-4c45-b2c6-a53c2f03ac10"). InnerVolumeSpecName "kube-api-access-4dd59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:05.226849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.226827    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-xks-ca" pod "3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" (UID: "3fb3fe42-9773-4c45-b2c6-a53c2f03ac10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:05.325215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.325140    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:05.325215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.325166    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dd59\" (UniqueName: \"kubernetes.io/projected/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-kube-api-access-4dd59\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:05.325215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.325176    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:05.684875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.684840    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"]
Oct 29 23:33:05.685484 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.685467    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" containerName="revoke-keys"
Oct 29 23:33:05.685535 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.685488    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" containerName="revoke-keys"
Oct 29 23:33:05.685603 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.685591    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" containerName="revoke-keys"
Oct 29 23:33:05.689254 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.689233    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.710866 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.710839    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"]
Oct 29 23:33:05.730464 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.730434    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/2813efdc-d715-4cac-9756-8b3ef952a4ec-kube-api-access-tnwlb\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.730622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.730482    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.730691 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.730667    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.831514 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.831479    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.831713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.831554    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/2813efdc-d715-4cac-9756-8b3ef952a4ec-kube-api-access-tnwlb\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.831713 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.831589    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.832228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.832208    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.833832 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.833814    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:05.853139 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:05.853112    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/2813efdc-d715-4cac-9756-8b3ef952a4ec-kube-api-access-tnwlb\") pod \"maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:06.004972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.004883    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:06.054392 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.054364    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"
Oct 29 23:33:06.054565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.054400    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp" event={"ID":"3fb3fe42-9773-4c45-b2c6-a53c2f03ac10","Type":"ContainerDied","Data":"96eafeffee03aad4e714ecf54d16e66045beb8a1bef5a541e8cdbdea5bf786b4"}
Oct 29 23:33:06.054565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.054451    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96eafeffee03aad4e714ecf54d16e66045beb8a1bef5a541e8cdbdea5bf786b4"
Oct 29 23:33:06.308789 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.308752    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"]
Oct 29 23:33:06.310126 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:06.310097    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2813efdc_d715_4cac_9756_8b3ef952a4ec.slice/crio-d8245ad1d6ecaf45eb151d609449405e0f15355072c71e70cbe3f515ed7e448b WatchSource:0}: Error finding container d8245ad1d6ecaf45eb151d609449405e0f15355072c71e70cbe3f515ed7e448b: Status 404 returned error can't find the container with id d8245ad1d6ecaf45eb151d609449405e0f15355072c71e70cbe3f515ed7e448b
Oct 29 23:33:06.791406 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.791373    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:33:06.846346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846312    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-certs\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846364    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-podinfo\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846413    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-kube-api-access-p8bx7\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-kube-api-access-p8bx7\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846567    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-credential-socket\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846626    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-data\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846637    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-certs" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.846688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846663    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-envoy\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846719    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istiod-ca-cert\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846757    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-token\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-token\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.846964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846834    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-credential-socket" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.846964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846856    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-data" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.846964 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.846871    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-socket\") pod \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\" (UID: \"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff\") "
Oct 29 23:33:06.851103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.847179    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istiod-ca-cert" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:06.851103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.847302    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.851103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.847321    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.851103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.847415    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.851103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.847432    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.851103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.847717    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-socket" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.854945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.854918    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-podinfo" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:06.854945 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.854936    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-token" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:06.855103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.854924    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-envoy" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.855103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.854955    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-kube-api-access-p8bx7" pod "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" (UID: "3ba31fd7-d55b-4305-8218-2d13ba7fa4ff"). InnerVolumeSpecName "kube-api-access-p8bx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:06.964861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.964741    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.964861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.964789    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.964861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.964804    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.964861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.964816    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.964861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:06.964830    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8bx7\" (UniqueName: \"kubernetes.io/projected/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff-kube-api-access-p8bx7\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:07.061516 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.061478    4827 generic.go:356] "Generic (PLEG): container finished" podID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerID="2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5" exitCode=0
Oct 29 23:33:07.061722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.061532    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"
Oct 29 23:33:07.061722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.061552    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" event={"ID":"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff","Type":"ContainerDied","Data":"2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5"}
Oct 29 23:33:07.061722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.061612    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn" event={"ID":"3ba31fd7-d55b-4305-8218-2d13ba7fa4ff","Type":"ContainerDied","Data":"0ca6392142aa139578750814ec6f9d4ed0f7075d51cb2e7d4204992e93acf1c1"}
Oct 29 23:33:07.061722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.061642    4827 scope.go:122] "RemoveContainer" containerID="2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5"
Oct 29 23:33:07.063345 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.063322    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9" event={"ID":"2813efdc-d715-4cac-9756-8b3ef952a4ec","Type":"ContainerStarted","Data":"6c39a9761141e477d75f40beaf212f1f0e11b7dcf794980b708dfad0a398d909"}
Oct 29 23:33:07.063523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.063502    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9" event={"ID":"2813efdc-d715-4cac-9756-8b3ef952a4ec","Type":"ContainerStarted","Data":"d8245ad1d6ecaf45eb151d609449405e0f15355072c71e70cbe3f515ed7e448b"}
Oct 29 23:33:07.073968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.073950    4827 scope.go:122] "RemoveContainer" containerID="8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db"
Oct 29 23:33:07.082748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.082729    4827 scope.go:122] "RemoveContainer" containerID="2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5"
Oct 29 23:33:07.083075 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:07.083055    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5\": container with ID starting with 2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5 not found: ID does not exist" containerID="2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5"
Oct 29 23:33:07.083142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.083083    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5"} err="failed to get container status \"2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5\": rpc error: code = NotFound desc = could not find container \"2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5\": container with ID starting with 2224dd880f8520ec18013a6eddc22c3ac1b53d577f34a4c250d49ef13f6965b5 not found: ID does not exist"
Oct 29 23:33:07.083142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.083102    4827 scope.go:122] "RemoveContainer" containerID="8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db"
Oct 29 23:33:07.083384 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:07.083365    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db\": container with ID starting with 8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db not found: ID does not exist" containerID="8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db"
Oct 29 23:33:07.083433 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.083396    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db"} err="failed to get container status \"8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db\": rpc error: code = NotFound desc = could not find container \"8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db\": container with ID starting with 8903b23798e25e6bdc0f893c6cf61b39e2aadc2d66352abf41f457feb84e70db not found: ID does not exist"
Oct 29 23:33:07.112359 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.112339    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"]
Oct 29 23:33:07.122844 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.122823    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-vpwwn"]
Oct 29 23:33:07.166412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.161432    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9" podStartSLOduration=2.161414512 podStartE2EDuration="2.161414512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:07.15344456 +0000 UTC m=+3022.776355582" watchObservedRunningTime="2026-10-29 23:33:07.161414512 +0000 UTC m=+3022.784325533"
Oct 29 23:33:07.256507 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.256437    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"]
Oct 29 23:33:07.259124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:07.259100    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e131af-fkwds"]
Oct 29 23:33:08.075098 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:08.075064    4827 generic.go:356] "Generic (PLEG): container finished" podID="2813efdc-d715-4cac-9756-8b3ef952a4ec" containerID="6c39a9761141e477d75f40beaf212f1f0e11b7dcf794980b708dfad0a398d909" exitCode=0
Oct 29 23:33:08.075658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:08.075134    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9" event={"ID":"2813efdc-d715-4cac-9756-8b3ef952a4ec","Type":"ContainerDied","Data":"6c39a9761141e477d75f40beaf212f1f0e11b7dcf794980b708dfad0a398d909"}
Oct 29 23:33:08.487780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:08.487736    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" path="/var/lib/kubelet/pods/3ba31fd7-d55b-4305-8218-2d13ba7fa4ff/volumes"
Oct 29 23:33:08.488194 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:08.488180    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="458cf807-b753-42c7-9519-44c292b0af5b" path="/var/lib/kubelet/pods/458cf807-b753-42c7-9519-44c292b0af5b/volumes"
Oct 29 23:33:09.243400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.243377    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:09.291090 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.291058    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-xks-ca\") pod \"2813efdc-d715-4cac-9756-8b3ef952a4ec\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") "
Oct 29 23:33:09.291262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.291196    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-openshift-ca\") pod \"2813efdc-d715-4cac-9756-8b3ef952a4ec\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") "
Oct 29 23:33:09.291262 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.291238    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2813efdc-d715-4cac-9756-8b3ef952a4ec-kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/2813efdc-d715-4cac-9756-8b3ef952a4ec-kube-api-access-tnwlb\") pod \"2813efdc-d715-4cac-9756-8b3ef952a4ec\" (UID: \"2813efdc-d715-4cac-9756-8b3ef952a4ec\") "
Oct 29 23:33:09.291573 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.291545    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-openshift-ca" pod "2813efdc-d715-4cac-9756-8b3ef952a4ec" (UID: "2813efdc-d715-4cac-9756-8b3ef952a4ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:09.291699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.291648    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:09.293380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.293350    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-xks-ca" pod "2813efdc-d715-4cac-9756-8b3ef952a4ec" (UID: "2813efdc-d715-4cac-9756-8b3ef952a4ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:09.293646 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.293626    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2813efdc-d715-4cac-9756-8b3ef952a4ec-kube-api-access-tnwlb" pod "2813efdc-d715-4cac-9756-8b3ef952a4ec" (UID: "2813efdc-d715-4cac-9756-8b3ef952a4ec"). InnerVolumeSpecName "kube-api-access-tnwlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:09.392518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.392489    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnwlb\" (UniqueName: \"kubernetes.io/projected/2813efdc-d715-4cac-9756-8b3ef952a4ec-kube-api-access-tnwlb\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:09.392518 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:09.392517    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2813efdc-d715-4cac-9756-8b3ef952a4ec-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:10.087498 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.087469    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"
Oct 29 23:33:10.087706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.087502    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9" event={"ID":"2813efdc-d715-4cac-9756-8b3ef952a4ec","Type":"ContainerDied","Data":"d8245ad1d6ecaf45eb151d609449405e0f15355072c71e70cbe3f515ed7e448b"}
Oct 29 23:33:10.087706 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.087562    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8245ad1d6ecaf45eb151d609449405e0f15355072c71e70cbe3f515ed7e448b"
Oct 29 23:33:10.341654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.341569    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"]
Oct 29 23:33:10.361439 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.361411    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"]
Oct 29 23:33:10.430894 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:10.430870    4827 logging.go:55] [core] [Channel #10271 SubChannel #10272]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.38:50052", ServerName: "10.134.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.38:50052: connect: connection refused"
Oct 29 23:33:10.431340 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:10.431318    4827 logging.go:55] [core] [Channel #10273 SubChannel #10274]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:50052", ServerName: "10.134.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:50052: connect: connection refused"
Oct 29 23:33:10.671132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.671107    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 23:33:10.705332 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.705309    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab7316e-15d7-41d9-9496-1d59d982b4d9-kube-api-access-j8k5q\" (UniqueName: \"kubernetes.io/projected/4ab7316e-15d7-41d9-9496-1d59d982b4d9-kube-api-access-j8k5q\") pod \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") "
Oct 29 23:33:10.705446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.705398    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab7316e-15d7-41d9-9496-1d59d982b4d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7316e-15d7-41d9-9496-1d59d982b4d9-plugins-config-volume\") pod \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\" (UID: \"4ab7316e-15d7-41d9-9496-1d59d982b4d9\") "
Oct 29 23:33:10.705758 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.705739    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab7316e-15d7-41d9-9496-1d59d982b4d9-plugins-config-volume" pod "4ab7316e-15d7-41d9-9496-1d59d982b4d9" (UID: "4ab7316e-15d7-41d9-9496-1d59d982b4d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:10.707488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.707380    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab7316e-15d7-41d9-9496-1d59d982b4d9-kube-api-access-j8k5q" pod "4ab7316e-15d7-41d9-9496-1d59d982b4d9" (UID: "4ab7316e-15d7-41d9-9496-1d59d982b4d9"). InnerVolumeSpecName "kube-api-access-j8k5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:10.719293 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.719270    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 23:33:10.806968 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.806860    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-plugins-config-volume\") pod \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") "
Oct 29 23:33:10.844587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.807004    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-kube-api-access-cxx22\" (UniqueName: \"kubernetes.io/projected/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-kube-api-access-cxx22\") pod \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\" (UID: \"a90a2c0c-652b-4f6b-9f19-efff8dff39fb\") "
Oct 29 23:33:10.844587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.807208    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-plugins-config-volume" pod "a90a2c0c-652b-4f6b-9f19-efff8dff39fb" (UID: "a90a2c0c-652b-4f6b-9f19-efff8dff39fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:10.844587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.807288    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8k5q\" (UniqueName: \"kubernetes.io/projected/4ab7316e-15d7-41d9-9496-1d59d982b4d9-kube-api-access-j8k5q\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:10.844587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.807303    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:10.844587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.807313    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7316e-15d7-41d9-9496-1d59d982b4d9-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:10.844587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.808966    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-kube-api-access-cxx22" pod "a90a2c0c-652b-4f6b-9f19-efff8dff39fb" (UID: "a90a2c0c-652b-4f6b-9f19-efff8dff39fb"). InnerVolumeSpecName "kube-api-access-cxx22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:10.908130 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:10.908055    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxx22\" (UniqueName: \"kubernetes.io/projected/a90a2c0c-652b-4f6b-9f19-efff8dff39fb-kube-api-access-cxx22\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:11.094959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.094926    4827 generic.go:356] "Generic (PLEG): container finished" podID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerID="4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0" exitCode=0
Oct 29 23:33:11.095142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.094979    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"
Oct 29 23:33:11.095142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.095011    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" event={"ID":"4ab7316e-15d7-41d9-9496-1d59d982b4d9","Type":"ContainerDied","Data":"4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0"}
Oct 29 23:33:11.095142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.095075    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" event={"ID":"4ab7316e-15d7-41d9-9496-1d59d982b4d9","Type":"ContainerDied","Data":"118956687d172f729913ae7bb777effd85b6ed96589c9f836eba45de5640b587"}
Oct 29 23:33:11.095142 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.095100    4827 scope.go:122] "RemoveContainer" containerID="4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0"
Oct 29 23:33:11.096400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.096376    4827 generic.go:356] "Generic (PLEG): container finished" podID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerID="d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813" exitCode=0
Oct 29 23:33:11.096492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.096413    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"
Oct 29 23:33:11.096544 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.096421    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" event={"ID":"a90a2c0c-652b-4f6b-9f19-efff8dff39fb","Type":"ContainerDied","Data":"d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813"}
Oct 29 23:33:11.096602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.096550    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" event={"ID":"a90a2c0c-652b-4f6b-9f19-efff8dff39fb","Type":"ContainerDied","Data":"417817a51786e93a3116a11b63b49c120f6da0ce3bb8f4e723dc78e46fdd5ade"}
Oct 29 23:33:11.105298 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.105275    4827 scope.go:122] "RemoveContainer" containerID="4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0"
Oct 29 23:33:11.105679 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:11.105648    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0\": container with ID starting with 4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0 not found: ID does not exist" containerID="4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0"
Oct 29 23:33:11.105759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.105690    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0"} err="failed to get container status \"4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0\": rpc error: code = NotFound desc = could not find container \"4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0\": container with ID starting with 4c1367443ac82d1b1117e24b424b3d3124c85a6298bf8a12181b772d3d7b93f0 not found: ID does not exist"
Oct 29 23:33:11.105759 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.105716    4827 scope.go:122] "RemoveContainer" containerID="d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813"
Oct 29 23:33:11.122350 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.122326    4827 scope.go:122] "RemoveContainer" containerID="d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813"
Oct 29 23:33:11.122719 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:11.122684    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813\": container with ID starting with d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813 not found: ID does not exist" containerID="d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813"
Oct 29 23:33:11.122846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.122734    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813"} err="failed to get container status \"d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813\": rpc error: code = NotFound desc = could not find container \"d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813\": container with ID starting with d64f31b1925cb2a42f31e530b41abdb80e8de093332acee524069134dfda9813 not found: ID does not exist"
Oct 29 23:33:11.156849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.156819    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"]
Oct 29 23:33:11.160374 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.160317    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf"]
Oct 29 23:33:11.191478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.191452    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"]
Oct 29 23:33:11.197442 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.197418    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v"]
Oct 29 23:33:11.430808 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.430721    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e131af-65fcd4b947-bdh9v" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.38:50052\" within 1s: context deadline exceeded"
Oct 29 23:33:11.431154 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.431001    4827 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e131af-5c44d7d7fd-mfflf" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:50052\" within 1s: context deadline exceeded"
Oct 29 23:33:11.896577 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.896550    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:33:11.914336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:11.914308    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:33:12.020367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.020293    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-kube-api-access-shqhr\" (UniqueName: \"kubernetes.io/projected/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-kube-api-access-shqhr\") pod \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") "
Oct 29 23:33:12.020367 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.020352    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-plugins-config-volume\") pod \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\" (UID: \"fd4199f4-e699-464a-9d1d-5c23ba2cc72e\") "
Oct 29 23:33:12.020586 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.020455    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-kube-api-access-lxrl9\" (UniqueName: \"kubernetes.io/projected/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-kube-api-access-lxrl9\") pod \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") "
Oct 29 23:33:12.020586 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.020501    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-plugins-config-volume\") pod \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\" (UID: \"afa6dcd1-b9cb-41e9-bc66-4bdc29478492\") "
Oct 29 23:33:12.020729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.020664    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-plugins-config-volume" pod "fd4199f4-e699-464a-9d1d-5c23ba2cc72e" (UID: "fd4199f4-e699-464a-9d1d-5c23ba2cc72e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:12.020829 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.020812    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:12.020975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.020957    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-plugins-config-volume" pod "afa6dcd1-b9cb-41e9-bc66-4bdc29478492" (UID: "afa6dcd1-b9cb-41e9-bc66-4bdc29478492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:12.022396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.022376    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-kube-api-access-shqhr" pod "fd4199f4-e699-464a-9d1d-5c23ba2cc72e" (UID: "fd4199f4-e699-464a-9d1d-5c23ba2cc72e"). InnerVolumeSpecName "kube-api-access-shqhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:12.022522 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.022503    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-kube-api-access-lxrl9" pod "afa6dcd1-b9cb-41e9-bc66-4bdc29478492" (UID: "afa6dcd1-b9cb-41e9-bc66-4bdc29478492"). InnerVolumeSpecName "kube-api-access-lxrl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:12.107843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.107804    4827 generic.go:356] "Generic (PLEG): container finished" podID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" containerID="d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780" exitCode=0
Oct 29 23:33:12.108043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.107845    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"
Oct 29 23:33:12.108043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.107854    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" event={"ID":"afa6dcd1-b9cb-41e9-bc66-4bdc29478492","Type":"ContainerDied","Data":"d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780"}
Oct 29 23:33:12.108043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.107917    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7" event={"ID":"afa6dcd1-b9cb-41e9-bc66-4bdc29478492","Type":"ContainerDied","Data":"2d2affb8d05d0b3cfacb8ad7e0ee6657f6279c208e24eeb9277c308f5205cba1"}
Oct 29 23:33:12.108043 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.107946    4827 scope.go:122] "RemoveContainer" containerID="d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780"
Oct 29 23:33:12.109540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.109517    4827 generic.go:356] "Generic (PLEG): container finished" podID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" containerID="b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf" exitCode=0
Oct 29 23:33:12.109642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.109572    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" event={"ID":"fd4199f4-e699-464a-9d1d-5c23ba2cc72e","Type":"ContainerDied","Data":"b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf"}
Oct 29 23:33:12.109642 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.109609    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn" event={"ID":"fd4199f4-e699-464a-9d1d-5c23ba2cc72e","Type":"ContainerDied","Data":"1869fc6fe1b2f5d419c1483d7818dac8725f40826bd652e172fa398d7b5d4d19"}
Oct 29 23:33:12.109727 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.109672    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"
Oct 29 23:33:12.118923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.118908    4827 scope.go:122] "RemoveContainer" containerID="d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780"
Oct 29 23:33:12.119179 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:12.119159    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780\": container with ID starting with d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780 not found: ID does not exist" containerID="d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780"
Oct 29 23:33:12.119280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.119187    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780"} err="failed to get container status \"d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780\": rpc error: code = NotFound desc = could not find container \"d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780\": container with ID starting with d7ee3b61d680d858263c4ebd02de2c81d08636826daed7bfbac324d3d8d74780 not found: ID does not exist"
Oct 29 23:33:12.119280 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.119201    4827 scope.go:122] "RemoveContainer" containerID="b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf"
Oct 29 23:33:12.121276 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.121258    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxrl9\" (UniqueName: \"kubernetes.io/projected/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-kube-api-access-lxrl9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:12.121357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.121277    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa6dcd1-b9cb-41e9-bc66-4bdc29478492-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:12.121357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.121287    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqhr\" (UniqueName: \"kubernetes.io/projected/fd4199f4-e699-464a-9d1d-5c23ba2cc72e-kube-api-access-shqhr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:12.127360 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.127345    4827 scope.go:122] "RemoveContainer" containerID="b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf"
Oct 29 23:33:12.127623 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:12.127606    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf\": container with ID starting with b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf not found: ID does not exist" containerID="b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf"
Oct 29 23:33:12.127675 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.127628    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf"} err="failed to get container status \"b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf\": rpc error: code = NotFound desc = could not find container \"b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf\": container with ID starting with b3266688ad162bdc2f957354515001be8a53733dd22b3645e9050bac14feb2cf not found: ID does not exist"
Oct 29 23:33:12.154956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.154936    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"]
Oct 29 23:33:12.161732 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.161710    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4504f1-6799d666d4-z65bn"]
Oct 29 23:33:12.192895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.192862    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"]
Oct 29 23:33:12.197623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.197599    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4504f1-7bb6d8d7d7-lkrr7"]
Oct 29 23:33:12.490336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.490301    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" path="/var/lib/kubelet/pods/4ab7316e-15d7-41d9-9496-1d59d982b4d9/volumes"
Oct 29 23:33:12.490889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.490871    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" path="/var/lib/kubelet/pods/a90a2c0c-652b-4f6b-9f19-efff8dff39fb/volumes"
Oct 29 23:33:12.491264 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.491246    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" path="/var/lib/kubelet/pods/afa6dcd1-b9cb-41e9-bc66-4bdc29478492/volumes"
Oct 29 23:33:12.491648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:12.491631    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" path="/var/lib/kubelet/pods/fd4199f4-e699-464a-9d1d-5c23ba2cc72e/volumes"
Oct 29 23:33:14.722669 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:14.722637    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"]
Oct 29 23:33:14.723159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:14.722900    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy" containerID="cri-o://a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5" gracePeriod=30
Oct 29 23:33:17.821708 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:17.821678    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"]
Oct 29 23:33:17.824434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:17.824413    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-67363b-hmqlp"]
Oct 29 23:33:18.487441 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:18.487408    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb3fe42-9773-4c45-b2c6-a53c2f03ac10" path="/var/lib/kubelet/pods/3fb3fe42-9773-4c45-b2c6-a53c2f03ac10/volumes"
Oct 29 23:33:19.986479 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:19.986455    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 23:33:20.113032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.112961    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-certs\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113017    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-data\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113052    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-kube-api-access-c7jf9\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-kube-api-access-c7jf9\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113149    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-credential-socket\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113195    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istiod-ca-cert\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113242 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113231    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-token\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-token\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113268    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-socket\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113275    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-data" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:20.113473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113289    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-certs" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:20.113473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113301    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-envoy\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113337    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-podinfo\") pod \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\" (UID: \"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96\") "
Oct 29 23:33:20.113782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113477    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-credential-socket" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:20.113782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113488    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-socket" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:20.113782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113581    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istiod-ca-cert" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:20.113782 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113741    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.114026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113761    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.114026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113810    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.114026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113826    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.114026 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.113839    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.115419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.115389    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-token" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:20.115563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.115422    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-podinfo" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:20.115563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.115455    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-kube-api-access-c7jf9" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "kube-api-access-c7jf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:20.115563 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.115459    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-envoy" pod "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" (UID: "f4b0d56f-4958-442c-9c39-c9d7b5a1fd96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:20.161695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.161667    4827 generic.go:356] "Generic (PLEG): container finished" podID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerID="a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5" exitCode=0
Oct 29 23:33:20.161904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.161717    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"
Oct 29 23:33:20.161904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.161754    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" event={"ID":"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96","Type":"ContainerDied","Data":"a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5"}
Oct 29 23:33:20.161904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.161847    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv" event={"ID":"f4b0d56f-4958-442c-9c39-c9d7b5a1fd96","Type":"ContainerDied","Data":"b3d0b1b7c0020b854131af2fa50cb32b7273563dc931ac82647f65dd72d42772"}
Oct 29 23:33:20.161904 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.161879    4827 scope.go:122] "RemoveContainer" containerID="a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5"
Oct 29 23:33:20.171542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.171520    4827 scope.go:122] "RemoveContainer" containerID="fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f"
Oct 29 23:33:20.180550 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.180533    4827 scope.go:122] "RemoveContainer" containerID="a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5"
Oct 29 23:33:20.180826 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:20.180807    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5\": container with ID starting with a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5 not found: ID does not exist" containerID="a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5"
Oct 29 23:33:20.180902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.180834    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5"} err="failed to get container status \"a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5\": rpc error: code = NotFound desc = could not find container \"a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5\": container with ID starting with a38cb324cf6a47f4ab454787c3d969fec5ae12fb234c8d1d618ade7278ecefd5 not found: ID does not exist"
Oct 29 23:33:20.180902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.180851    4827 scope.go:122] "RemoveContainer" containerID="fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f"
Oct 29 23:33:20.181074 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:20.181054    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f\": container with ID starting with fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f not found: ID does not exist" containerID="fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f"
Oct 29 23:33:20.181135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.181084    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f"} err="failed to get container status \"fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f\": rpc error: code = NotFound desc = could not find container \"fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f\": container with ID starting with fb502e8a32bf629a72f3a5d06243f0cb4a954265abe6daba7aff987155a6729f not found: ID does not exist"
Oct 29 23:33:20.208627 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.208599    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"]
Oct 29 23:33:20.213941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.213915    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-9lzdv"]
Oct 29 23:33:20.214376 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.214357    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7jf9\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-kube-api-access-c7jf9\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.214421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.214384    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.214421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.214394    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.214421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.214402    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:20.486313 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:20.486279    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" path="/var/lib/kubelet/pods/f4b0d56f-4958-442c-9c39-c9d7b5a1fd96/volumes"
Oct 29 23:33:21.167703 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.167671    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"]
Oct 29 23:33:21.168129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168083    4827 cpu_manager.go:414] "containerMap: removing container" podUID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" containerName="payload-pre-processing"
Oct 29 23:33:21.168129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168098    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" containerName="payload-pre-processing"
Oct 29 23:33:21.168129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168107    4827 cpu_manager.go:414] "containerMap: removing container" podUID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" containerName="payload-processing"
Oct 29 23:33:21.168129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168113    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" containerName="payload-processing"
Oct 29 23:33:21.168129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168124    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy"
Oct 29 23:33:21.168129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168128    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168137    4827 cpu_manager.go:414] "containerMap: removing container" podUID="2813efdc-d715-4cac-9756-8b3ef952a4ec" containerName="revoke-keys"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168142    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2813efdc-d715-4cac-9756-8b3ef952a4ec" containerName="revoke-keys"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168149    4827 cpu_manager.go:414] "containerMap: removing container" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerName="payload-pre-processing"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168153    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerName="payload-pre-processing"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168167    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerName="payload-processing"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168172    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerName="payload-processing"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168177    4827 cpu_manager.go:414] "containerMap: removing container" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168182    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168189    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168193    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168200    4827 cpu_manager.go:414] "containerMap: removing container" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168204    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168255    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab7316e-15d7-41d9-9496-1d59d982b4d9" containerName="payload-processing"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168262    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168267    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168272    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba31fd7-d55b-4305-8218-2d13ba7fa4ff" containerName="istio-proxy"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168281    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa6dcd1-b9cb-41e9-bc66-4bdc29478492" containerName="payload-pre-processing"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168286    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4199f4-e699-464a-9d1d-5c23ba2cc72e" containerName="payload-processing"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168293    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="2813efdc-d715-4cac-9756-8b3ef952a4ec" containerName="revoke-keys"
Oct 29 23:33:21.168393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.168298    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="a90a2c0c-652b-4f6b-9f19-efff8dff39fb" containerName="payload-pre-processing"
Oct 29 23:33:21.172875 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.172850    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.189124 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.189096    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:33:21.196878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.196851    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:33:21.199647 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.199620    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"]
Oct 29 23:33:21.201565 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.201545    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"]
Oct 29 23:33:21.201984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.201970    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b0d56f-4958-442c-9c39-c9d7b5a1fd96" containerName="istio-proxy"
Oct 29 23:33:21.205045 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.205025    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.219177 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.219136    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"]
Oct 29 23:33:21.324799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.324733    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.325002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.324812    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbkf\" (UniqueName: \"kubernetes.io/projected/0d510b90-dedc-43b7-94da-1805c4dd9ff5-kube-api-access-trbkf\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.325002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.324916    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.325002 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.324944    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trgv\" (UniqueName: \"kubernetes.io/projected/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-kube-api-access-4trgv\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.325190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.325088    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.325249 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.325201    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.426735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.426648    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.426735 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.426718    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.426993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.426750    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbkf\" (UniqueName: \"kubernetes.io/projected/0d510b90-dedc-43b7-94da-1805c4dd9ff5-kube-api-access-trbkf\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.426993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.426838    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.426993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.426863    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trgv\" (UniqueName: \"kubernetes.io/projected/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-kube-api-access-4trgv\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.426993 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.426945    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.427554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.427525    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.427676 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.427567    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.429258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.429237    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.429258 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.429247    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.448492 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.448467    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbkf\" (UniqueName: \"kubernetes.io/projected/0d510b90-dedc-43b7-94da-1805c4dd9ff5-kube-api-access-trbkf\") pod \"maas-api-revoke-sub-d3a6107f-s9tx2\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.448746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.448727    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trgv\" (UniqueName: \"kubernetes.io/projected/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-kube-api-access-4trgv\") pod \"maas-api-revoke-sub-7ebcfc9e-9zd8v\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.485623 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.485599    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:21.524738 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.524706    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:21.784644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.784621    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"]
Oct 29 23:33:21.786282 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:21.786257    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d510b90_dedc_43b7_94da_1805c4dd9ff5.slice/crio-99922bc577555e07682f3623db3aafbaa7d6a91516831c89203aaf33c101d3e1 WatchSource:0}: Error finding container 99922bc577555e07682f3623db3aafbaa7d6a91516831c89203aaf33c101d3e1: Status 404 returned error can't find the container with id 99922bc577555e07682f3623db3aafbaa7d6a91516831c89203aaf33c101d3e1
Oct 29 23:33:21.849394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:21.849373    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"]
Oct 29 23:33:21.850842 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:21.850816    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce60e1d_21f2_43a7_a6fa_436e5aea7cb0.slice/crio-425f36a3b03b6ae23b7381c2e89dc84e7c06452f2e3e6f4e8a697b2634921ec7 WatchSource:0}: Error finding container 425f36a3b03b6ae23b7381c2e89dc84e7c06452f2e3e6f4e8a697b2634921ec7: Status 404 returned error can't find the container with id 425f36a3b03b6ae23b7381c2e89dc84e7c06452f2e3e6f4e8a697b2634921ec7
Oct 29 23:33:22.195977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:22.194036    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v" event={"ID":"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0","Type":"ContainerStarted","Data":"425f36a3b03b6ae23b7381c2e89dc84e7c06452f2e3e6f4e8a697b2634921ec7"}
Oct 29 23:33:22.196859 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:22.196741    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2" event={"ID":"0d510b90-dedc-43b7-94da-1805c4dd9ff5","Type":"ContainerStarted","Data":"99922bc577555e07682f3623db3aafbaa7d6a91516831c89203aaf33c101d3e1"}
Oct 29 23:33:23.204318 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:23.204274    4827 generic.go:356] "Generic (PLEG): container finished" podID="dce60e1d-21f2-43a7-a6fa-436e5aea7cb0" containerID="9ab18a896acf2f8808f31905d73b4e0fed56b6ba429f2fe52c29780a22fd4d29" exitCode=0
Oct 29 23:33:23.204742 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:23.204336    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v" event={"ID":"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0","Type":"ContainerDied","Data":"9ab18a896acf2f8808f31905d73b4e0fed56b6ba429f2fe52c29780a22fd4d29"}
Oct 29 23:33:23.206166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:23.206138    4827 generic.go:356] "Generic (PLEG): container finished" podID="0d510b90-dedc-43b7-94da-1805c4dd9ff5" containerID="64bbe696149bc91b97dcd9a17d0a9c23cd9c8b778f08fcf512d15d2e6f148308" exitCode=0
Oct 29 23:33:23.206308 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:23.206189    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2" event={"ID":"0d510b90-dedc-43b7-94da-1805c4dd9ff5","Type":"ContainerDied","Data":"64bbe696149bc91b97dcd9a17d0a9c23cd9c8b778f08fcf512d15d2e6f148308"}
Oct 29 23:33:24.355095 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.355071    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:24.370806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.370781    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:24.458207 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458166    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-xks-ca\") pod \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") "
Oct 29 23:33:24.458420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458234    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-xks-ca\") pod \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") "
Oct 29 23:33:24.458420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458276    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-kube-api-access-4trgv\" (UniqueName: \"kubernetes.io/projected/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-kube-api-access-4trgv\") pod \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") "
Oct 29 23:33:24.458420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458338    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-openshift-ca\") pod \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\" (UID: \"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0\") "
Oct 29 23:33:24.458420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458408    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d510b90-dedc-43b7-94da-1805c4dd9ff5-kube-api-access-trbkf\" (UniqueName: \"kubernetes.io/projected/0d510b90-dedc-43b7-94da-1805c4dd9ff5-kube-api-access-trbkf\") pod \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") "
Oct 29 23:33:24.458650 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458480    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-openshift-ca\") pod \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\" (UID: \"0d510b90-dedc-43b7-94da-1805c4dd9ff5\") "
Oct 29 23:33:24.458834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458797    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-openshift-ca" pod "dce60e1d-21f2-43a7-a6fa-436e5aea7cb0" (UID: "dce60e1d-21f2-43a7-a6fa-436e5aea7cb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:24.459028 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.458898    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:24.459116 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.459036    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-openshift-ca" pod "0d510b90-dedc-43b7-94da-1805c4dd9ff5" (UID: "0d510b90-dedc-43b7-94da-1805c4dd9ff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:24.460553 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.460526    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-xks-ca" pod "0d510b90-dedc-43b7-94da-1805c4dd9ff5" (UID: "0d510b90-dedc-43b7-94da-1805c4dd9ff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:24.460665 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.460595    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-kube-api-access-4trgv" pod "dce60e1d-21f2-43a7-a6fa-436e5aea7cb0" (UID: "dce60e1d-21f2-43a7-a6fa-436e5aea7cb0"). InnerVolumeSpecName "kube-api-access-4trgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:24.460892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.460866    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-xks-ca" pod "dce60e1d-21f2-43a7-a6fa-436e5aea7cb0" (UID: "dce60e1d-21f2-43a7-a6fa-436e5aea7cb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:24.460892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.460880    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d510b90-dedc-43b7-94da-1805c4dd9ff5-kube-api-access-trbkf" pod "0d510b90-dedc-43b7-94da-1805c4dd9ff5" (UID: "0d510b90-dedc-43b7-94da-1805c4dd9ff5"). InnerVolumeSpecName "kube-api-access-trbkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:24.560252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.560217    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:24.560252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.560246    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:24.560434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.560262    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d510b90-dedc-43b7-94da-1805c4dd9ff5-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:24.560434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.560275    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trgv\" (UniqueName: \"kubernetes.io/projected/dce60e1d-21f2-43a7-a6fa-436e5aea7cb0-kube-api-access-4trgv\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:24.560434 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:24.560287    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trbkf\" (UniqueName: \"kubernetes.io/projected/0d510b90-dedc-43b7-94da-1805c4dd9ff5-kube-api-access-trbkf\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.219467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.219433    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2"
Oct 29 23:33:25.219677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.219439    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a6107f-s9tx2" event={"ID":"0d510b90-dedc-43b7-94da-1805c4dd9ff5","Type":"ContainerDied","Data":"99922bc577555e07682f3623db3aafbaa7d6a91516831c89203aaf33c101d3e1"}
Oct 29 23:33:25.219677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.219607    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99922bc577555e07682f3623db3aafbaa7d6a91516831c89203aaf33c101d3e1"
Oct 29 23:33:25.220815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.220792    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v"
Oct 29 23:33:25.220938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.220811    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcfc9e-9zd8v" event={"ID":"dce60e1d-21f2-43a7-a6fa-436e5aea7cb0","Type":"ContainerDied","Data":"425f36a3b03b6ae23b7381c2e89dc84e7c06452f2e3e6f4e8a697b2634921ec7"}
Oct 29 23:33:25.220938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.220853    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="425f36a3b03b6ae23b7381c2e89dc84e7c06452f2e3e6f4e8a697b2634921ec7"
Oct 29 23:33:25.296834 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.296802    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"]
Oct 29 23:33:25.297048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.297024    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy" containerID="cri-o://c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6" gracePeriod=30
Oct 29 23:33:25.361604 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.361553    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main" containerID="cri-o://7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca" gracePeriod=45
Oct 29 23:33:25.382587 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.382557    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main" containerID="cri-o://ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af" gracePeriod=45
Oct 29 23:33:25.630839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.630814    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:33:25.649224 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.649202    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:33:25.782864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.782790    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0f92f1-7de0-4ba3-a324-415985263db5-kube-api-access-kf7gs\" (UniqueName: \"kubernetes.io/projected/4a0f92f1-7de0-4ba3-a324-415985263db5-kube-api-access-kf7gs\") pod \"4a0f92f1-7de0-4ba3-a324-415985263db5\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") "
Oct 29 23:33:25.782864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.782847    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-home\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-home\") pod \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") "
Oct 29 23:33:25.783068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.782880    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a0f92f1-7de0-4ba3-a324-415985263db5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f92f1-7de0-4ba3-a324-415985263db5-tls-certs\") pod \"4a0f92f1-7de0-4ba3-a324-415985263db5\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") "
Oct 29 23:33:25.783068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.782905    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tls-certs\") pod \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") "
Oct 29 23:33:25.783068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.782932    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-home\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-home\") pod \"4a0f92f1-7de0-4ba3-a324-415985263db5\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") "
Oct 29 23:33:25.783068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.782986    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-tmp-dir\") pod \"4a0f92f1-7de0-4ba3-a324-415985263db5\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") "
Oct 29 23:33:25.783068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783038    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-model-cache\") pod \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") "
Oct 29 23:33:25.783068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783067    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-model-cache\") pod \"4a0f92f1-7de0-4ba3-a324-415985263db5\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") "
Oct 29 23:33:25.783430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783114    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-dshm\") pod \"4a0f92f1-7de0-4ba3-a324-415985263db5\" (UID: \"4a0f92f1-7de0-4ba3-a324-415985263db5\") "
Oct 29 23:33:25.783430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783154    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712e715a-7d5a-41df-a6a9-7d04f90c07b8-kube-api-access-2745b\" (UniqueName: \"kubernetes.io/projected/712e715a-7d5a-41df-a6a9-7d04f90c07b8-kube-api-access-2745b\") pod \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") "
Oct 29 23:33:25.783430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783153    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-home" pod "712e715a-7d5a-41df-a6a9-7d04f90c07b8" (UID: "712e715a-7d5a-41df-a6a9-7d04f90c07b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.783430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783204    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-dshm\") pod \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") "
Oct 29 23:33:25.783430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783231    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tmp-dir\") pod \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\" (UID: \"712e715a-7d5a-41df-a6a9-7d04f90c07b8\") "
Oct 29 23:33:25.783430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783245    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-home" pod "4a0f92f1-7de0-4ba3-a324-415985263db5" (UID: "4a0f92f1-7de0-4ba3-a324-415985263db5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.783430 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783370    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-model-cache" pod "4a0f92f1-7de0-4ba3-a324-415985263db5" (UID: "4a0f92f1-7de0-4ba3-a324-415985263db5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.783868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783470    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-tmp-dir" pod "4a0f92f1-7de0-4ba3-a324-415985263db5" (UID: "4a0f92f1-7de0-4ba3-a324-415985263db5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.783868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783617    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-model-cache" pod "712e715a-7d5a-41df-a6a9-7d04f90c07b8" (UID: "712e715a-7d5a-41df-a6a9-7d04f90c07b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.783868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783733    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.783868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783750    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.783868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783795    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.783868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783808    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.783868 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.783844    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.784260 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.784018    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tmp-dir" pod "712e715a-7d5a-41df-a6a9-7d04f90c07b8" (UID: "712e715a-7d5a-41df-a6a9-7d04f90c07b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.785628 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.785593    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0f92f1-7de0-4ba3-a324-415985263db5-tls-certs" pod "4a0f92f1-7de0-4ba3-a324-415985263db5" (UID: "4a0f92f1-7de0-4ba3-a324-415985263db5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:25.785751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.785659    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tls-certs" pod "712e715a-7d5a-41df-a6a9-7d04f90c07b8" (UID: "712e715a-7d5a-41df-a6a9-7d04f90c07b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:25.785751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.785671    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0f92f1-7de0-4ba3-a324-415985263db5-kube-api-access-kf7gs" pod "4a0f92f1-7de0-4ba3-a324-415985263db5" (UID: "4a0f92f1-7de0-4ba3-a324-415985263db5"). InnerVolumeSpecName "kube-api-access-kf7gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:25.785751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.785672    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-dshm" pod "712e715a-7d5a-41df-a6a9-7d04f90c07b8" (UID: "712e715a-7d5a-41df-a6a9-7d04f90c07b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.785751 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.785740    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-dshm" pod "4a0f92f1-7de0-4ba3-a324-415985263db5" (UID: "4a0f92f1-7de0-4ba3-a324-415985263db5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:25.785985 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.785970    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712e715a-7d5a-41df-a6a9-7d04f90c07b8-kube-api-access-2745b" pod "712e715a-7d5a-41df-a6a9-7d04f90c07b8" (UID: "712e715a-7d5a-41df-a6a9-7d04f90c07b8"). InnerVolumeSpecName "kube-api-access-2745b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:25.884872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.884837    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a0f92f1-7de0-4ba3-a324-415985263db5-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.884872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.884866    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2745b\" (UniqueName: \"kubernetes.io/projected/712e715a-7d5a-41df-a6a9-7d04f90c07b8-kube-api-access-2745b\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.884872 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.884879    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.885088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.884889    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.885088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.884905    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7gs\" (UniqueName: \"kubernetes.io/projected/4a0f92f1-7de0-4ba3-a324-415985263db5-kube-api-access-kf7gs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.885088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.884913    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a0f92f1-7de0-4ba3-a324-415985263db5-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:25.885088 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:25.884922    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e715a-7d5a-41df-a6a9-7d04f90c07b8-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:26.034372 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034295    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"]
Oct 29 23:33:26.034672 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034660    4827 cpu_manager.go:414] "containerMap: removing container" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main"
Oct 29 23:33:26.034719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034673    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main"
Oct 29 23:33:26.034719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034690    4827 cpu_manager.go:414] "containerMap: removing container" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main"
Oct 29 23:33:26.034719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034695    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main"
Oct 29 23:33:26.034719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034705    4827 cpu_manager.go:414] "containerMap: removing container" podUID="dce60e1d-21f2-43a7-a6fa-436e5aea7cb0" containerName="revoke-keys"
Oct 29 23:33:26.034719 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034710    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce60e1d-21f2-43a7-a6fa-436e5aea7cb0" containerName="revoke-keys"
Oct 29 23:33:26.034912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034721    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0d510b90-dedc-43b7-94da-1805c4dd9ff5" containerName="revoke-keys"
Oct 29 23:33:26.034912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034726    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d510b90-dedc-43b7-94da-1805c4dd9ff5" containerName="revoke-keys"
Oct 29 23:33:26.034912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034813    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerName="main"
Oct 29 23:33:26.034912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034823    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerName="main"
Oct 29 23:33:26.034912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034832    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce60e1d-21f2-43a7-a6fa-436e5aea7cb0" containerName="revoke-keys"
Oct 29 23:33:26.034912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.034838    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d510b90-dedc-43b7-94da-1805c4dd9ff5" containerName="revoke-keys"
Oct 29 23:33:26.038955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.038933    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.057195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.057168    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:33:26.057322 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.057252    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:33:26.059938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.059916    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"]
Oct 29 23:33:26.188230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.188196    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.188230 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.188236    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69n8r\" (UniqueName: \"kubernetes.io/projected/7957dcc7-0557-41ca-bfc0-231ff59122ba-kube-api-access-69n8r\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.188557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.188341    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.229201 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.229170    4827 generic.go:356] "Generic (PLEG): container finished" podID="4a0f92f1-7de0-4ba3-a324-415985263db5" containerID="ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af" exitCode=0
Oct 29 23:33:26.229474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.229222    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"
Oct 29 23:33:26.229474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.229244    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" event={"ID":"4a0f92f1-7de0-4ba3-a324-415985263db5","Type":"ContainerDied","Data":"ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af"}
Oct 29 23:33:26.229474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.229281    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8" event={"ID":"4a0f92f1-7de0-4ba3-a324-415985263db5","Type":"ContainerDied","Data":"09e47fad63844474767ca91b2c87557b98141baacb432bb218309c909749a659"}
Oct 29 23:33:26.229474 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.229301    4827 scope.go:122] "RemoveContainer" containerID="ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af"
Oct 29 23:33:26.230938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.230911    4827 generic.go:356] "Generic (PLEG): container finished" podID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" containerID="7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca" exitCode=0
Oct 29 23:33:26.231086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.230950    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" event={"ID":"712e715a-7d5a-41df-a6a9-7d04f90c07b8","Type":"ContainerDied","Data":"7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca"}
Oct 29 23:33:26.231086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.230979    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8" event={"ID":"712e715a-7d5a-41df-a6a9-7d04f90c07b8","Type":"ContainerDied","Data":"4d70dc3035503f7b075b7b6aa5db485a95fd323e095f87299612933d087b177d"}
Oct 29 23:33:26.231086 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.230956    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"
Oct 29 23:33:26.238651 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.238631    4827 scope.go:122] "RemoveContainer" containerID="ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af"
Oct 29 23:33:26.238958 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:26.238939    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af\": container with ID starting with ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af not found: ID does not exist" containerID="ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af"
Oct 29 23:33:26.239030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.238965    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af"} err="failed to get container status \"ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af\": rpc error: code = NotFound desc = could not find container \"ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af\": container with ID starting with ae20ae31959dc8df3ab40c77175a362d49daf23d294b584ab1d822f40f4366af not found: ID does not exist"
Oct 29 23:33:26.239030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.238982    4827 scope.go:122] "RemoveContainer" containerID="7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca"
Oct 29 23:33:26.248128 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.248112    4827 scope.go:122] "RemoveContainer" containerID="7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca"
Oct 29 23:33:26.248411 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:26.248392    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca\": container with ID starting with 7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca not found: ID does not exist" containerID="7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca"
Oct 29 23:33:26.248478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.248419    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca"} err="failed to get container status \"7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca\": rpc error: code = NotFound desc = could not find container \"7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca\": container with ID starting with 7b61c853cd1d4e0a468f0ed21f599d262dd49fb265ab560349ed0cc7486c36ca not found: ID does not exist"
Oct 29 23:33:26.277256 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.277229    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"]
Oct 29 23:33:26.280444 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.280421    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/simulator-w4-063b46-kserve-5b46c8fb64-q5vl8"]
Oct 29 23:33:26.289478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.289403    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.289478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.289461    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.289666 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.289497    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69n8r\" (UniqueName: \"kubernetes.io/projected/7957dcc7-0557-41ca-bfc0-231ff59122ba-kube-api-access-69n8r\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.290129 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.290106    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.291935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.291914    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.311807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.311739    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69n8r\" (UniqueName: \"kubernetes.io/projected/7957dcc7-0557-41ca-bfc0-231ff59122ba-kube-api-access-69n8r\") pod \"maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.312966 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.312944    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"]
Oct 29 23:33:26.316747 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.316726    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-w4-063b46-kserve-78b64b9fd8-dtrf8"]
Oct 29 23:33:26.352556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.352536    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:26.486191 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.486159    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0f92f1-7de0-4ba3-a324-415985263db5" path="/var/lib/kubelet/pods/4a0f92f1-7de0-4ba3-a324-415985263db5/volumes"
Oct 29 23:33:26.486581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.486567    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712e715a-7d5a-41df-a6a9-7d04f90c07b8" path="/var/lib/kubelet/pods/712e715a-7d5a-41df-a6a9-7d04f90c07b8/volumes"
Oct 29 23:33:26.667212 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:26.667155    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"]
Oct 29 23:33:27.238975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:27.238876    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p" event={"ID":"7957dcc7-0557-41ca-bfc0-231ff59122ba","Type":"ContainerStarted","Data":"f78eec211906b388cfe2c9a099f394a02b8c8dffecc3ffff2bf50abde35eeaad"}
Oct 29 23:33:27.238975 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:27.238949    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p" event={"ID":"7957dcc7-0557-41ca-bfc0-231ff59122ba","Type":"ContainerStarted","Data":"52a4e918c191c0ee9782f7fd132109c0008e7aa2d91187cc5a323636df8ae571"}
Oct 29 23:33:27.276017 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:27.275965    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p" podStartSLOduration=1.275953364 podStartE2EDuration="1.275953364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:27.275710911 +0000 UTC m=+3042.898621930" watchObservedRunningTime="2026-10-29 23:33:27.275953364 +0000 UTC m=+3042.898864384"
Oct 29 23:33:28.248977 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:28.248942    4827 generic.go:356] "Generic (PLEG): container finished" podID="7957dcc7-0557-41ca-bfc0-231ff59122ba" containerID="f78eec211906b388cfe2c9a099f394a02b8c8dffecc3ffff2bf50abde35eeaad" exitCode=0
Oct 29 23:33:28.249396 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:28.249012    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p" event={"ID":"7957dcc7-0557-41ca-bfc0-231ff59122ba","Type":"ContainerDied","Data":"f78eec211906b388cfe2c9a099f394a02b8c8dffecc3ffff2bf50abde35eeaad"}
Oct 29 23:33:29.385446 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.385420    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:29.522749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.522663    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-openshift-ca\") pod \"7957dcc7-0557-41ca-bfc0-231ff59122ba\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") "
Oct 29 23:33:29.522749 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.522717    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7957dcc7-0557-41ca-bfc0-231ff59122ba-kube-api-access-69n8r\" (UniqueName: \"kubernetes.io/projected/7957dcc7-0557-41ca-bfc0-231ff59122ba-kube-api-access-69n8r\") pod \"7957dcc7-0557-41ca-bfc0-231ff59122ba\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") "
Oct 29 23:33:29.522983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.522849    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-xks-ca\") pod \"7957dcc7-0557-41ca-bfc0-231ff59122ba\" (UID: \"7957dcc7-0557-41ca-bfc0-231ff59122ba\") "
Oct 29 23:33:29.523068 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.523045    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-openshift-ca" pod "7957dcc7-0557-41ca-bfc0-231ff59122ba" (UID: "7957dcc7-0557-41ca-bfc0-231ff59122ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:29.523159 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.523146    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:29.524849 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.524825    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-xks-ca" pod "7957dcc7-0557-41ca-bfc0-231ff59122ba" (UID: "7957dcc7-0557-41ca-bfc0-231ff59122ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:29.524972 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.524861    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7957dcc7-0557-41ca-bfc0-231ff59122ba-kube-api-access-69n8r" pod "7957dcc7-0557-41ca-bfc0-231ff59122ba" (UID: "7957dcc7-0557-41ca-bfc0-231ff59122ba"). InnerVolumeSpecName "kube-api-access-69n8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:29.624523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.624484    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7957dcc7-0557-41ca-bfc0-231ff59122ba-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:29.624523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:29.624518    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69n8r\" (UniqueName: \"kubernetes.io/projected/7957dcc7-0557-41ca-bfc0-231ff59122ba-kube-api-access-69n8r\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.265271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.265243    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"
Oct 29 23:33:30.265473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.265278    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p" event={"ID":"7957dcc7-0557-41ca-bfc0-231ff59122ba","Type":"ContainerDied","Data":"52a4e918c191c0ee9782f7fd132109c0008e7aa2d91187cc5a323636df8ae571"}
Oct 29 23:33:30.265473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.265337    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a4e918c191c0ee9782f7fd132109c0008e7aa2d91187cc5a323636df8ae571"
Oct 29 23:33:30.549228 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.549199    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:33:30.734625 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734594    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-envoy\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.734850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734641    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-kube-api-access-n55ws\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-kube-api-access-n55ws\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.734850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734676    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de570536-b5be-4262-b005-499d5113ae21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de570536-b5be-4262-b005-499d5113ae21-istiod-ca-cert\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.734850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734696    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-istio-token\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-istio-token\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.734850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734721    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-data\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.734850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734797    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de570536-b5be-4262-b005-499d5113ae21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de570536-b5be-4262-b005-499d5113ae21-istio-podinfo\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.734850 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734845    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-socket\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.735217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734880    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-certs\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.735217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.734928    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-credential-socket\") pod \"de570536-b5be-4262-b005-499d5113ae21\" (UID: \"de570536-b5be-4262-b005-499d5113ae21\") "
Oct 29 23:33:30.735335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.735220    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-socket" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:30.735335 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.735246    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de570536-b5be-4262-b005-499d5113ae21-istiod-ca-cert" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:30.735461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.735425    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-data" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:30.735461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.735451    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-credential-socket" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:30.735555 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.735488    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-certs" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:30.736996 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.736971    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-envoy" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:30.737103 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.737016    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-istio-token" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:30.737273 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.737258    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de570536-b5be-4262-b005-499d5113ae21-istio-podinfo" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:30.737315 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.737300    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-kube-api-access-n55ws" pod "de570536-b5be-4262-b005-499d5113ae21" (UID: "de570536-b5be-4262-b005-499d5113ae21"). InnerVolumeSpecName "kube-api-access-n55ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:30.780729 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.780659    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"]
Oct 29 23:33:30.835956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.835931    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.835956 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.835955    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55ws\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-kube-api-access-n55ws\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.836152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.835965    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de570536-b5be-4262-b005-499d5113ae21-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.836152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.835975    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de570536-b5be-4262-b005-499d5113ae21-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.836152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.835982    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.836152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.835990    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de570536-b5be-4262-b005-499d5113ae21-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.836152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.835997    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.836152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.836005    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:30.836152 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:30.836012    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de570536-b5be-4262-b005-499d5113ae21-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:31.272380 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.272347    4827 generic.go:356] "Generic (PLEG): container finished" podID="de570536-b5be-4262-b005-499d5113ae21" containerID="c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6" exitCode=0
Oct 29 23:33:31.272599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.272403    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"
Oct 29 23:33:31.272599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.272441    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" event={"ID":"de570536-b5be-4262-b005-499d5113ae21","Type":"ContainerDied","Data":"c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6"}
Oct 29 23:33:31.272599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.272506    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z" event={"ID":"de570536-b5be-4262-b005-499d5113ae21","Type":"ContainerDied","Data":"7539d033b90fca91ef0951a52325ecf87a8fdd18182b2973c23ab0453612b636"}
Oct 29 23:33:31.272599 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.272536    4827 scope.go:122] "RemoveContainer" containerID="c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6"
Oct 29 23:33:31.284023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.284002    4827 scope.go:122] "RemoveContainer" containerID="c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6"
Oct 29 23:33:31.284318 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:31.284298    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6\": container with ID starting with c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6 not found: ID does not exist" containerID="c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6"
Oct 29 23:33:31.284391 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.284331    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6"} err="failed to get container status \"c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6\": rpc error: code = NotFound desc = could not find container \"c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6\": container with ID starting with c2eb4181e0aa9840b3c26c5898063d32accace28fb0f0e386847bdbef3cc57b6 not found: ID does not exist"
Oct 29 23:33:31.330922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.330895    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"]
Oct 29 23:33:31.340914 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:31.340888    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-g8v8z"]
Oct 29 23:33:32.486961 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:32.486933    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de570536-b5be-4262-b005-499d5113ae21" path="/var/lib/kubelet/pods/de570536-b5be-4262-b005-499d5113ae21/volumes"
Oct 29 23:33:32.823632 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:32.823548    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:33:36.507381 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:36.507349    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"]
Oct 29 23:33:38.545208 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:38.545171    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"]
Oct 29 23:33:38.556486 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:38.556449    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-063b46-xpnm9"]
Oct 29 23:33:39.859449 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.859416    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"]
Oct 29 23:33:39.859891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.859855    4827 cpu_manager.go:414] "containerMap: removing container" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy"
Oct 29 23:33:39.859891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.859870    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy"
Oct 29 23:33:39.859891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.859882    4827 cpu_manager.go:414] "containerMap: removing container" podUID="7957dcc7-0557-41ca-bfc0-231ff59122ba" containerName="revoke-keys"
Oct 29 23:33:39.859891 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.859888    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7957dcc7-0557-41ca-bfc0-231ff59122ba" containerName="revoke-keys"
Oct 29 23:33:39.860048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.859935    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="de570536-b5be-4262-b005-499d5113ae21" containerName="istio-proxy"
Oct 29 23:33:39.860048 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.859943    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="7957dcc7-0557-41ca-bfc0-231ff59122ba" containerName="revoke-keys"
Oct 29 23:33:39.864481 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.864459    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:39.879957 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.879935    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"]
Oct 29 23:33:39.883488 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.883469    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:39.889828 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.889729    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:33:39.895112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.890156    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\""
Oct 29 23:33:39.895112 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.893432    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"]
Oct 29 23:33:39.923508 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.923481    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"]
Oct 29 23:33:39.929979 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.929952    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgw9m\" (UniqueName: \"kubernetes.io/projected/616da088-a8a8-4779-8803-524aa43cebb2-kube-api-access-jgw9m\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:39.930118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.930070    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:39.930118 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.930098    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:39.930245 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.930218    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dgdq\" (UniqueName: \"kubernetes.io/projected/41879f6b-c67d-48e6-88e6-7adca8234186-kube-api-access-8dgdq\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:39.930339 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.930317    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:39.930404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:39.930367    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.031688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.031651    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgw9m\" (UniqueName: \"kubernetes.io/projected/616da088-a8a8-4779-8803-524aa43cebb2-kube-api-access-jgw9m\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:40.031923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.031731    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:40.031923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.031756    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.031923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.031830    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dgdq\" (UniqueName: \"kubernetes.io/projected/41879f6b-c67d-48e6-88e6-7adca8234186-kube-api-access-8dgdq\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.031923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.031877    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:40.031923 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.031915    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.032616 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.032586    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:40.032740 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.032672    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.034299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.034271    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.034399 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.034340    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:40.053500 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.053470    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dgdq\" (UniqueName: \"kubernetes.io/projected/41879f6b-c67d-48e6-88e6-7adca8234186-kube-api-access-8dgdq\") pod \"maas-api-revoke-sub-ff1b00de-4fz7f\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.053680 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.053659    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgw9m\" (UniqueName: \"kubernetes.io/projected/616da088-a8a8-4779-8803-524aa43cebb2-kube-api-access-jgw9m\") pod \"maas-api-revoke-sub-b3c8d76d-mf7jx\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:40.177379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.177342    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:40.213683 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.213652    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:40.477310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.477279    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"]
Oct 29 23:33:40.479232 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:40.479202    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616da088_a8a8_4779_8803_524aa43cebb2.slice/crio-37d9cdc2e9eacb28181e06c875c625b29ecc619a227e39f05c7022b73da23765 WatchSource:0}: Error finding container 37d9cdc2e9eacb28181e06c875c625b29ecc619a227e39f05c7022b73da23765: Status 404 returned error can't find the container with id 37d9cdc2e9eacb28181e06c875c625b29ecc619a227e39f05c7022b73da23765
Oct 29 23:33:40.486821 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.486795    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2813efdc-d715-4cac-9756-8b3ef952a4ec" path="/var/lib/kubelet/pods/2813efdc-d715-4cac-9756-8b3ef952a4ec/volumes"
Oct 29 23:33:40.516734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:40.516705    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"]
Oct 29 23:33:40.520913 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:33:40.520807    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41879f6b_c67d_48e6_88e6_7adca8234186.slice/crio-d7d356631585ccfe4ccb5012cad890e98b59377a3c61b1a96371bfeaab59939c WatchSource:0}: Error finding container d7d356631585ccfe4ccb5012cad890e98b59377a3c61b1a96371bfeaab59939c: Status 404 returned error can't find the container with id d7d356631585ccfe4ccb5012cad890e98b59377a3c61b1a96371bfeaab59939c
Oct 29 23:33:41.362440 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.362402    4827 generic.go:356] "Generic (PLEG): container finished" podID="41879f6b-c67d-48e6-88e6-7adca8234186" containerID="bbfc2e7aef8c330ee6499fdc29adaa3fccd31627584d46c5f7278bee0b3e7cc6" exitCode=0
Oct 29 23:33:41.362920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.362512    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f" event={"ID":"41879f6b-c67d-48e6-88e6-7adca8234186","Type":"ContainerDied","Data":"bbfc2e7aef8c330ee6499fdc29adaa3fccd31627584d46c5f7278bee0b3e7cc6"}
Oct 29 23:33:41.362920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.362562    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f" event={"ID":"41879f6b-c67d-48e6-88e6-7adca8234186","Type":"ContainerStarted","Data":"d7d356631585ccfe4ccb5012cad890e98b59377a3c61b1a96371bfeaab59939c"}
Oct 29 23:33:41.364865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.364828    4827 generic.go:356] "Generic (PLEG): container finished" podID="616da088-a8a8-4779-8803-524aa43cebb2" containerID="767443a0577082e86e406d2a37c2bf5136ec641eb0d2f582a7b210d78fb7c11a" exitCode=0
Oct 29 23:33:41.365019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.364887    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx" event={"ID":"616da088-a8a8-4779-8803-524aa43cebb2","Type":"ContainerDied","Data":"767443a0577082e86e406d2a37c2bf5136ec641eb0d2f582a7b210d78fb7c11a"}
Oct 29 23:33:41.365019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.364929    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx" event={"ID":"616da088-a8a8-4779-8803-524aa43cebb2","Type":"ContainerStarted","Data":"37d9cdc2e9eacb28181e06c875c625b29ecc619a227e39f05c7022b73da23765"}
Oct 29 23:33:41.507792 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.507746    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs" podUID="9cc1f75d-3bef-4805-9e22-b48a8d0c1099" containerName="payload-processing" containerID="cri-o://f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2" gracePeriod=25
Oct 29 23:33:41.751185 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.751161    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 23:33:41.772451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.772376    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-plugins-config-volume\") pod \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") "
Oct 29 23:33:41.772451 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.772418    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-kube-api-access-sscg6\" (UniqueName: \"kubernetes.io/projected/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-kube-api-access-sscg6\") pod \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\" (UID: \"9cc1f75d-3bef-4805-9e22-b48a8d0c1099\") "
Oct 29 23:33:41.772745 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.772724    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-plugins-config-volume" pod "9cc1f75d-3bef-4805-9e22-b48a8d0c1099" (UID: "9cc1f75d-3bef-4805-9e22-b48a8d0c1099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:41.774475 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.774457    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-kube-api-access-sscg6" pod "9cc1f75d-3bef-4805-9e22-b48a8d0c1099" (UID: "9cc1f75d-3bef-4805-9e22-b48a8d0c1099"). InnerVolumeSpecName "kube-api-access-sscg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:41.873087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.873052    4827 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-plugins-config-volume\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:41.873087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:41.873084    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscg6\" (UniqueName: \"kubernetes.io/projected/9cc1f75d-3bef-4805-9e22-b48a8d0c1099-kube-api-access-sscg6\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:42.371874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.371788    4827 generic.go:356] "Generic (PLEG): container finished" podID="9cc1f75d-3bef-4805-9e22-b48a8d0c1099" containerID="f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2" exitCode=0
Oct 29 23:33:42.371874 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.371823    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs" event={"ID":"9cc1f75d-3bef-4805-9e22-b48a8d0c1099","Type":"ContainerDied","Data":"f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2"}
Oct 29 23:33:42.372418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.371871    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"
Oct 29 23:33:42.372418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.371894    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs" event={"ID":"9cc1f75d-3bef-4805-9e22-b48a8d0c1099","Type":"ContainerDied","Data":"8c51c5065df6541fee5b574f98d60a214ae36bf2c8620ce9c11e1ad075613037"}
Oct 29 23:33:42.372418 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.371926    4827 scope.go:122] "RemoveContainer" containerID="f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2"
Oct 29 23:33:42.383711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.383686    4827 scope.go:122] "RemoveContainer" containerID="f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2"
Oct 29 23:33:42.384112 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:42.384094    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2\": container with ID starting with f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2 not found: ID does not exist" containerID="f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2"
Oct 29 23:33:42.384166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.384124    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2"} err="failed to get container status \"f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2\": rpc error: code = NotFound desc = could not find container \"f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2\": container with ID starting with f77c7170754ebdc42a854eb1bc858716deecdac2eee004106820b447fc968dd2 not found: ID does not exist"
Oct 29 23:33:42.417215 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.417180    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"]
Oct 29 23:33:42.428319 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.428291    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-08e15d-6f5566b97b-gh8hs"]
Oct 29 23:33:42.488941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.488905    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc1f75d-3bef-4805-9e22-b48a8d0c1099" path="/var/lib/kubelet/pods/9cc1f75d-3bef-4805-9e22-b48a8d0c1099/volumes"
Oct 29 23:33:42.523917 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.523896    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:42.525930 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.525915    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:42.579631 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.579588    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616da088-a8a8-4779-8803-524aa43cebb2-kube-api-access-jgw9m\" (UniqueName: \"kubernetes.io/projected/616da088-a8a8-4779-8803-524aa43cebb2-kube-api-access-jgw9m\") pod \"616da088-a8a8-4779-8803-524aa43cebb2\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") "
Oct 29 23:33:42.579787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.579657    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-openshift-ca\") pod \"41879f6b-c67d-48e6-88e6-7adca8234186\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") "
Oct 29 23:33:42.579787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.579704    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-xks-ca\") pod \"616da088-a8a8-4779-8803-524aa43cebb2\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") "
Oct 29 23:33:42.579913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.579824    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-openshift-ca\") pod \"616da088-a8a8-4779-8803-524aa43cebb2\" (UID: \"616da088-a8a8-4779-8803-524aa43cebb2\") "
Oct 29 23:33:42.579913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.579901    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41879f6b-c67d-48e6-88e6-7adca8234186-kube-api-access-8dgdq\" (UniqueName: \"kubernetes.io/projected/41879f6b-c67d-48e6-88e6-7adca8234186-kube-api-access-8dgdq\") pod \"41879f6b-c67d-48e6-88e6-7adca8234186\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") "
Oct 29 23:33:42.580024 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.579955    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-xks-ca\") pod \"41879f6b-c67d-48e6-88e6-7adca8234186\" (UID: \"41879f6b-c67d-48e6-88e6-7adca8234186\") "
Oct 29 23:33:42.580217 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.580189    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-openshift-ca" pod "616da088-a8a8-4779-8803-524aa43cebb2" (UID: "616da088-a8a8-4779-8803-524aa43cebb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:42.580382 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.580356    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:42.580453 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.580375    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-openshift-ca" pod "41879f6b-c67d-48e6-88e6-7adca8234186" (UID: "41879f6b-c67d-48e6-88e6-7adca8234186"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:42.582029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.581998    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616da088-a8a8-4779-8803-524aa43cebb2-kube-api-access-jgw9m" pod "616da088-a8a8-4779-8803-524aa43cebb2" (UID: "616da088-a8a8-4779-8803-524aa43cebb2"). InnerVolumeSpecName "kube-api-access-jgw9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:42.582253 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.582229    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-xks-ca" pod "41879f6b-c67d-48e6-88e6-7adca8234186" (UID: "41879f6b-c67d-48e6-88e6-7adca8234186"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:42.582325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.582305    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-xks-ca" pod "616da088-a8a8-4779-8803-524aa43cebb2" (UID: "616da088-a8a8-4779-8803-524aa43cebb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:42.582388 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.582318    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41879f6b-c67d-48e6-88e6-7adca8234186-kube-api-access-8dgdq" pod "41879f6b-c67d-48e6-88e6-7adca8234186" (UID: "41879f6b-c67d-48e6-88e6-7adca8234186"). InnerVolumeSpecName "kube-api-access-8dgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:42.681056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.681025    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-openshift-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:42.681056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.681052    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616da088-a8a8-4779-8803-524aa43cebb2-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:42.681056 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.681064    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dgdq\" (UniqueName: \"kubernetes.io/projected/41879f6b-c67d-48e6-88e6-7adca8234186-kube-api-access-8dgdq\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:42.681304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.681072    4827 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41879f6b-c67d-48e6-88e6-7adca8234186-maas-api-xks-ca\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:42.681304 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:42.681081    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgw9m\" (UniqueName: \"kubernetes.io/projected/616da088-a8a8-4779-8803-524aa43cebb2-kube-api-access-jgw9m\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:43.378995 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:43.378968    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f"
Oct 29 23:33:43.379437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:43.378962    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1b00de-4fz7f" event={"ID":"41879f6b-c67d-48e6-88e6-7adca8234186","Type":"ContainerDied","Data":"d7d356631585ccfe4ccb5012cad890e98b59377a3c61b1a96371bfeaab59939c"}
Oct 29 23:33:43.379437 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:43.379094    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d356631585ccfe4ccb5012cad890e98b59377a3c61b1a96371bfeaab59939c"
Oct 29 23:33:43.381379 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:43.381358    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx"
Oct 29 23:33:43.381501 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:43.381386    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c8d76d-mf7jx" event={"ID":"616da088-a8a8-4779-8803-524aa43cebb2","Type":"ContainerDied","Data":"37d9cdc2e9eacb28181e06c875c625b29ecc619a227e39f05c7022b73da23765"}
Oct 29 23:33:43.381501 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:43.381421    4827 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37d9cdc2e9eacb28181e06c875c625b29ecc619a227e39f05c7022b73da23765"
Oct 29 23:33:45.800502 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:45.800447    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main" containerID="cri-o://d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52" gracePeriod=45
Oct 29 23:33:46.045291 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.045266    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:33:46.111151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111061    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-model-cache\") pod \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") "
Oct 29 23:33:46.111151 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111137    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f9fb36-7afe-4473-8a19-b7b8106fff3f-kube-api-access-b82cg\" (UniqueName: \"kubernetes.io/projected/77f9fb36-7afe-4473-8a19-b7b8106fff3f-kube-api-access-b82cg\") pod \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") "
Oct 29 23:33:46.111476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111227    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-dshm\") pod \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") "
Oct 29 23:33:46.111476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111267    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-home\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-home\") pod \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") "
Oct 29 23:33:46.111476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111312    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-model-cache" pod "77f9fb36-7afe-4473-8a19-b7b8106fff3f" (UID: "77f9fb36-7afe-4473-8a19-b7b8106fff3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.111476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111361    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tls-certs\") pod \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") "
Oct 29 23:33:46.111476 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111424    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tmp-dir\") pod \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\" (UID: \"77f9fb36-7afe-4473-8a19-b7b8106fff3f\") "
Oct 29 23:33:46.111803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111536    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-home" pod "77f9fb36-7afe-4473-8a19-b7b8106fff3f" (UID: "77f9fb36-7afe-4473-8a19-b7b8106fff3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.111803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111744    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tmp-dir" pod "77f9fb36-7afe-4473-8a19-b7b8106fff3f" (UID: "77f9fb36-7afe-4473-8a19-b7b8106fff3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.111934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111821    4827 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-model-cache\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.111934 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.111839    4827 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-home\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.113654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.113628    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tls-certs" pod "77f9fb36-7afe-4473-8a19-b7b8106fff3f" (UID: "77f9fb36-7afe-4473-8a19-b7b8106fff3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:46.113863 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.113838    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-dshm" pod "77f9fb36-7afe-4473-8a19-b7b8106fff3f" (UID: "77f9fb36-7afe-4473-8a19-b7b8106fff3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.113954 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.113838    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f9fb36-7afe-4473-8a19-b7b8106fff3f-kube-api-access-b82cg" pod "77f9fb36-7afe-4473-8a19-b7b8106fff3f" (UID: "77f9fb36-7afe-4473-8a19-b7b8106fff3f"). InnerVolumeSpecName "kube-api-access-b82cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:46.212926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.212888    4827 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-dshm\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.212926 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.212920    4827 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tls-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.213135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.212959    4827 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f9fb36-7afe-4473-8a19-b7b8106fff3f-tmp-dir\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.213135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.212972    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82cg\" (UniqueName: \"kubernetes.io/projected/77f9fb36-7afe-4473-8a19-b7b8106fff3f-kube-api-access-b82cg\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.400076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.400043    4827 generic.go:356] "Generic (PLEG): container finished" podID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerID="d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52" exitCode=0
Oct 29 23:33:46.400285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.400093    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" event={"ID":"77f9fb36-7afe-4473-8a19-b7b8106fff3f","Type":"ContainerDied","Data":"d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52"}
Oct 29 23:33:46.400285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.400097    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"
Oct 29 23:33:46.400285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.400124    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6" event={"ID":"77f9fb36-7afe-4473-8a19-b7b8106fff3f","Type":"ContainerDied","Data":"a0e06de4acc62395eaba8abb1c4684b0792a4d4379c266a97a5817aae6b49adf"}
Oct 29 23:33:46.400285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.400140    4827 scope.go:122] "RemoveContainer" containerID="d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52"
Oct 29 23:33:46.409635 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.409620    4827 scope.go:122] "RemoveContainer" containerID="d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52"
Oct 29 23:33:46.409943 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:33:46.409923    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52\": container with ID starting with d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52 not found: ID does not exist" containerID="d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52"
Oct 29 23:33:46.410013 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.409950    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52"} err="failed to get container status \"d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52\": rpc error: code = NotFound desc = could not find container \"d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52\": container with ID starting with d46703facd8fb63425833450bd6a315b564336191f80d2cf10c268543a9d0e52 not found: ID does not exist"
Oct 29 23:33:46.448283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.448258    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"]
Oct 29 23:33:46.458780 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.458747    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-2-w0-08e15d-kserve-5bf88798df-vd2k6"]
Oct 29 23:33:46.485819 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:46.485762    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" path="/var/lib/kubelet/pods/77f9fb36-7afe-4473-8a19-b7b8106fff3f/volumes"
Oct 29 23:33:51.615352 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:51.615317    4827 scope.go:122] "RemoveContainer" containerID="e06fb27ac17a5e4813ba83a65b6a606664bd0df9eb04a8cf035189349aa0db63"
Oct 29 23:33:55.598704 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:55.598642    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"]
Oct 29 23:33:55.609044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:55.609014    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-08e15d-nsz2p"]
Oct 29 23:33:56.485815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:56.485786    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7957dcc7-0557-41ca-bfc0-231ff59122ba" path="/var/lib/kubelet/pods/7957dcc7-0557-41ca-bfc0-231ff59122ba/volumes"
Oct 29 23:33:59.032431 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.032391    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"]
Oct 29 23:33:59.050940 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.050864    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66056f9-tv27j"]
Oct 29 23:33:59.051149 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.050963    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"]
Oct 29 23:33:59.052385 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.052359    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffe4d3-7nzzb"]
Oct 29 23:33:59.692346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692316    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"]
Oct 29 23:33:59.692722 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692709    4827 cpu_manager.go:414] "containerMap: removing container" podUID="41879f6b-c67d-48e6-88e6-7adca8234186" containerName="revoke-keys"
Oct 29 23:33:59.692805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692724    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41879f6b-c67d-48e6-88e6-7adca8234186" containerName="revoke-keys"
Oct 29 23:33:59.692805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692742    4827 cpu_manager.go:414] "containerMap: removing container" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main"
Oct 29 23:33:59.692805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692748    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main"
Oct 29 23:33:59.692805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692775    4827 cpu_manager.go:414] "containerMap: removing container" podUID="9cc1f75d-3bef-4805-9e22-b48a8d0c1099" containerName="payload-processing"
Oct 29 23:33:59.692805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692782    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc1f75d-3bef-4805-9e22-b48a8d0c1099" containerName="payload-processing"
Oct 29 23:33:59.692805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692802    4827 cpu_manager.go:414] "containerMap: removing container" podUID="616da088-a8a8-4779-8803-524aa43cebb2" containerName="revoke-keys"
Oct 29 23:33:59.692805 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692807    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616da088-a8a8-4779-8803-524aa43cebb2" containerName="revoke-keys"
Oct 29 23:33:59.693038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692857    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc1f75d-3bef-4805-9e22-b48a8d0c1099" containerName="payload-processing"
Oct 29 23:33:59.693038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692870    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="616da088-a8a8-4779-8803-524aa43cebb2" containerName="revoke-keys"
Oct 29 23:33:59.693038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692879    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="41879f6b-c67d-48e6-88e6-7adca8234186" containerName="revoke-keys"
Oct 29 23:33:59.693038 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.692887    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f9fb36-7afe-4473-8a19-b7b8106fff3f" containerName="main"
Oct 29 23:33:59.698357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.698335    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.718895 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.718877    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-08e15d-openshift-default-dockercfg-kjhd8\""
Oct 29 23:33:59.732699 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.732674    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"]
Oct 29 23:33:59.858636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.858600    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.858889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.858647    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.858889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.858666    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.858889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.858775    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.858889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.858871    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lcr\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-kube-api-access-h9lcr\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.859123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.858981    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdae895f-57fd-4f6c-8842-150ff9b37757-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.859123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.859024    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.859123 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.859045    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.859287 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.859148    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.960947    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.960981    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961022 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961007    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961049    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lcr\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-kube-api-access-h9lcr\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961111    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdae895f-57fd-4f6c-8842-150ff9b37757-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961169    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961203    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961239    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961271    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961409    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961636 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961480    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961633    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961760 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961707    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdae895f-57fd-4f6c-8842-150ff9b37757-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.961942 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.961920    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.963519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.963496    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.963695 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.963679    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:33:59.995219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:33:59.995193    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:00.005202 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.005174    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lcr\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-kube-api-access-h9lcr\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:00.010865 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.010831    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:00.339166 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.339144    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"]
Oct 29 23:34:00.341328 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:34:00.341299    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdae895f_57fd_4f6c_8842_150ff9b37757.slice/crio-83b6056f483daaea40f27f03b18131737821d26bf4cfe08a02dd355106397875 WatchSource:0}: Error finding container 83b6056f483daaea40f27f03b18131737821d26bf4cfe08a02dd355106397875: Status 404 returned error can't find the container with id 83b6056f483daaea40f27f03b18131737821d26bf4cfe08a02dd355106397875
Oct 29 23:34:00.343472 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.343436    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:34:00.343581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.343502    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:34:00.343581 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.343531    4827 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 29 23:34:00.487004 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.486970    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e14136-c1d5-4f17-902c-f28aa8b2a182" path="/var/lib/kubelet/pods/91e14136-c1d5-4f17-902c-f28aa8b2a182/volumes"
Oct 29 23:34:00.487473 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.487456    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcefe833-486a-465f-8303-8ab70934cd96" path="/var/lib/kubelet/pods/fcefe833-486a-465f-8303-8ab70934cd96/volumes"
Oct 29 23:34:00.487761 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.487743    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5" event={"ID":"cdae895f-57fd-4f6c-8842-150ff9b37757","Type":"ContainerStarted","Data":"76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333"}
Oct 29 23:34:00.487824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.487803    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5" event={"ID":"cdae895f-57fd-4f6c-8842-150ff9b37757","Type":"ContainerStarted","Data":"83b6056f483daaea40f27f03b18131737821d26bf4cfe08a02dd355106397875"}
Oct 29 23:34:00.534795 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:00.533661    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5" podStartSLOduration=1.533641292 podStartE2EDuration="1.533641292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:00.53240212 +0000 UTC m=+3076.155313141" watchObservedRunningTime="2026-10-29 23:34:00.533641292 +0000 UTC m=+3076.156552314"
Oct 29 23:34:01.011803 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:01.011758    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:01.016841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:01.016818    4827 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:01.491715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:01.491685    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:01.492572 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:01.492551    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:02.763461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:02.763424    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"]
Oct 29 23:34:02.822076 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:34:02.822040    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:34:03.526226 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:03.526190    4827 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5" secret="" err="secret \"e2e-worker-w0-08e15d-openshift-default-dockercfg-kjhd8\" not found"
Oct 29 23:34:04.154554 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.154517    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"]
Oct 29 23:34:04.155127 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.154817    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy" containerID="cri-o://6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151" gracePeriod=30
Oct 29 23:34:04.165579 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.165553    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"]
Oct 29 23:34:04.165918 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.165874    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy" containerID="cri-o://3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c" gracePeriod=30
Oct 29 23:34:04.167580 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.167556    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cdfvp_9037b762-9013-436a-8abc-fc94d30b6f29/manager/0.log"
Oct 29 23:34:04.168190 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.168166    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"]
Oct 29 23:34:04.168395 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.168370    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy" containerID="cri-o://e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8" gracePeriod=30
Oct 29 23:34:04.531944 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:04.531851    4827 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5" podUID="cdae895f-57fd-4f6c-8842-150ff9b37757" containerName="istio-proxy" containerID="cri-o://76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333" gracePeriod=30
Oct 29 23:34:06.907044 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:06.907016    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:34:07.501898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:07.501867    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:34:08.109958 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:08.109925    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:34:09.411806 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.411753    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:34:09.452608 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.452585    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:34:09.455815 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.455791    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:34:09.556052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556026    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff7df262-3b46-4141-a339-bea7561b622f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7df262-3b46-4141-a339-bea7561b622f-istio-podinfo\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556064    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-istio-token\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-istio-token\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556096    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-data\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556123    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-credential-socket\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556196 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556156    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e63f57-51c7-426a-8967-3f8e1130f56d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e63f57-51c7-426a-8967-3f8e1130f56d-istiod-ca-cert\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556199    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-credential-socket\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556223    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-kube-api-access-bw6r2\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-kube-api-access-bw6r2\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556250    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-podinfo\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556266    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-token\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-token\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556293    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-data\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556326    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-kube-api-access-m9jvd\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-kube-api-access-m9jvd\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556343    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-certs\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556367    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-envoy\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556390 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556393    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-data\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556411    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istiod-ca-cert\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556440    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-envoy\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556490    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-certs\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556515    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-token\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-token\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556557    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-socket\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556585    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-kube-api-access-nvvx2\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-kube-api-access-nvvx2\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556610    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-podinfo\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556643    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7df262-3b46-4141-a339-bea7561b622f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7df262-3b46-4141-a339-bea7561b622f-istiod-ca-cert\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556697    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-credential-socket\") pod \"59e63f57-51c7-426a-8967-3f8e1130f56d\" (UID: \"59e63f57-51c7-426a-8967-3f8e1130f56d\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556731    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-socket\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556786    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-socket\") pod \"ff7df262-3b46-4141-a339-bea7561b622f\" (UID: \"ff7df262-3b46-4141-a339-bea7561b622f\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556813    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-certs\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.556938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.556857    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-envoy\") pod \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\" (UID: \"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b\") "
Oct 29 23:34:09.559602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559123    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-kube-api-access-m9jvd" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "kube-api-access-m9jvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:09.559602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559123    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-envoy" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.559602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559148    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-istio-token" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:09.559602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559352    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-certs" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.559602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559365    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-data" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.559602 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559491    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-kube-api-access-bw6r2" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "kube-api-access-bw6r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:09.560105 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559618    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-envoy" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.560105 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559728    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-data" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.560105 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.559970    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-credential-socket" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.560540 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.560294    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e63f57-51c7-426a-8967-3f8e1130f56d-istiod-ca-cert" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:09.560654 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.560625    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-credential-socket" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.560951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.560920    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff7df262-3b46-4141-a339-bea7561b622f-istio-podinfo" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:09.561300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.561274    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-certs" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.561728 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.561705    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-socket" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.561962 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.561931    4827 generic.go:356] "Generic (PLEG): container finished" podID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerID="6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151" exitCode=0
Oct 29 23:34:09.562071 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.562041    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" event={"ID":"59e63f57-51c7-426a-8967-3f8e1130f56d","Type":"ContainerDied","Data":"6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151"}
Oct 29 23:34:09.562148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.562090    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp" event={"ID":"59e63f57-51c7-426a-8967-3f8e1130f56d","Type":"ContainerDied","Data":"59b103b1a1d05777ecd964a50d1a0e8364adb79c3ec08113bc7fbffa1a41f21f"}
Oct 29 23:34:09.562148 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.562118    4827 scope.go:122] "RemoveContainer" containerID="6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151"
Oct 29 23:34:09.562283 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.562270    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"
Oct 29 23:34:09.562354 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.562316    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-kube-api-access-nvvx2" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "kube-api-access-nvvx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:09.562592 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.562562    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-podinfo" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:09.563136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563069    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-envoy" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.563240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563116    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-token" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:09.563240 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563085    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istiod-ca-cert" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:09.563677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563246    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-data" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.563677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563402    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-podinfo" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:09.563677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563444    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-credential-socket" pod "59e63f57-51c7-426a-8967-3f8e1130f56d" (UID: "59e63f57-51c7-426a-8967-3f8e1130f56d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.563677 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563537    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-socket" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.563933 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563703    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7df262-3b46-4141-a339-bea7561b622f-istiod-ca-cert" pod "ff7df262-3b46-4141-a339-bea7561b622f" (UID: "ff7df262-3b46-4141-a339-bea7561b622f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:09.563998 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.563919    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-socket" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.564058 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.564011    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-certs" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.564120 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.564097    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-token" pod "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" (UID: "0aa6ad85-8294-49a5-8c24-7d4fb9e1582b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:09.564807 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.564752    4827 generic.go:356] "Generic (PLEG): container finished" podID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerID="e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8" exitCode=0
Oct 29 23:34:09.564947 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.564919    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" event={"ID":"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b","Type":"ContainerDied","Data":"e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8"}
Oct 29 23:34:09.565020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.564944    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"
Oct 29 23:34:09.565020 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.564974    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b" event={"ID":"0aa6ad85-8294-49a5-8c24-7d4fb9e1582b","Type":"ContainerDied","Data":"ecd7f56fda81e80b599b45b2ca6575390ce155aad2cb97703f9b90b65d198290"}
Oct 29 23:34:09.566310 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.566291    4827 generic.go:356] "Generic (PLEG): container finished" podID="ff7df262-3b46-4141-a339-bea7561b622f" containerID="3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c" exitCode=0
Oct 29 23:34:09.566402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.566333    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"
Oct 29 23:34:09.566402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.566374    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" event={"ID":"ff7df262-3b46-4141-a339-bea7561b622f","Type":"ContainerDied","Data":"3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c"}
Oct 29 23:34:09.566510 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.566422    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5" event={"ID":"ff7df262-3b46-4141-a339-bea7561b622f","Type":"ContainerDied","Data":"02d26eadbad350597b53939ea20087e50677155904d46bceabbb84d7731461fe"}
Oct 29 23:34:09.637358 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.637333    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"]
Oct 29 23:34:09.644117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.644091    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8q72b"]
Oct 29 23:34:09.646111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.645992    4827 scope.go:122] "RemoveContainer" containerID="6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151"
Oct 29 23:34:09.647845 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:34:09.647814    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151\": container with ID starting with 6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151 not found: ID does not exist" containerID="6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151"
Oct 29 23:34:09.648421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.647858    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151"} err="failed to get container status \"6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151\": rpc error: code = NotFound desc = could not find container \"6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151\": container with ID starting with 6bcc7ff70eeb23673387f3384463149be9e737ca968eed1922ed99ba0f645151 not found: ID does not exist"
Oct 29 23:34:09.648421 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.647886    4827 scope.go:122] "RemoveContainer" containerID="e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8"
Oct 29 23:34:09.658064 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658039    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9jvd\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-kube-api-access-m9jvd\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658071    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658086    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658098    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658110    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658121    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658136 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658133    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658145    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658156    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658169    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvx2\" (UniqueName: \"kubernetes.io/projected/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-kube-api-access-nvvx2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658182    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658197    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7df262-3b46-4141-a339-bea7561b622f-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658209    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e63f57-51c7-426a-8967-3f8e1130f56d-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658222    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658234    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658247    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658260    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658271    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7df262-3b46-4141-a339-bea7561b622f-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658284    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658296    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658307    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658320    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e63f57-51c7-426a-8967-3f8e1130f56d-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658333    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658346    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw6r2\" (UniqueName: \"kubernetes.io/projected/ff7df262-3b46-4141-a339-bea7561b622f-kube-api-access-bw6r2\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658360    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658371    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e63f57-51c7-426a-8967-3f8e1130f56d-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.658466 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.658383    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7df262-3b46-4141-a339-bea7561b622f-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:09.659892 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.659871    4827 scope.go:122] "RemoveContainer" containerID="e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8"
Oct 29 23:34:09.660200 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:34:09.660183    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8\": container with ID starting with e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8 not found: ID does not exist" containerID="e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8"
Oct 29 23:34:09.660271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.660210    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8"} err="failed to get container status \"e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8\": rpc error: code = NotFound desc = could not find container \"e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8\": container with ID starting with e72e2c9d3f07cd1984c5fc149d06d7beb83a01a90936f3ceda333d90ac8eb9d8 not found: ID does not exist"
Oct 29 23:34:09.660271 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.660227    4827 scope.go:122] "RemoveContainer" containerID="3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c"
Oct 29 23:34:09.671117 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.671098    4827 scope.go:122] "RemoveContainer" containerID="3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c"
Oct 29 23:34:09.671479 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:34:09.671455    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c\": container with ID starting with 3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c not found: ID does not exist" containerID="3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c"
Oct 29 23:34:09.671707 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.671510    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c"} err="failed to get container status \"3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c\": rpc error: code = NotFound desc = could not find container \"3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c\": container with ID starting with 3e402d1aa33aeedae0f3a850a520ba6b2c0145a0d7caf37a408c0e64618a5e2c not found: ID does not exist"
Oct 29 23:34:09.674229 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.674206    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"]
Oct 29 23:34:09.679600 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.679569    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n6pr5"]
Oct 29 23:34:09.780867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.780843    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:09.904867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.904839    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"]
Oct 29 23:34:09.909861 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.909834    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k2lxp"]
Oct 29 23:34:09.960730 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.960696    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-podinfo\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.960902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.960785    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-token\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-token\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.960902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.960835    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-socket\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.960902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.960868    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-certs\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.961093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.960912    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdae895f-57fd-4f6c-8842-150ff9b37757-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdae895f-57fd-4f6c-8842-150ff9b37757-istiod-ca-cert\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.961093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.960953    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-envoy\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.961093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.960983    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-credential-socket\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.961093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.961037    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-kube-api-access-h9lcr\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-kube-api-access-h9lcr\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.961093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.961083    4827 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-data\") pod \"cdae895f-57fd-4f6c-8842-150ff9b37757\" (UID: \"cdae895f-57fd-4f6c-8842-150ff9b37757\") "
Oct 29 23:34:09.961413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.961382    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-credential-socket" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.961557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.961533    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdae895f-57fd-4f6c-8842-150ff9b37757-istiod-ca-cert" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:09.961557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.961540    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-data" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.961557 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.961551    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-socket" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.961785 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.961503    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-certs" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.963023 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.962970    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-podinfo" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:09.963113 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.963026    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-envoy" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:09.963299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.963278    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-token" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:09.963420 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:09.963363    4827 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-kube-api-access-h9lcr" pod "cdae895f-57fd-4f6c-8842-150ff9b37757" (UID: "cdae895f-57fd-4f6c-8842-150ff9b37757"). InnerVolumeSpecName "kube-api-access-h9lcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:10.062093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062063    4827 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-data\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062088    4827 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-podinfo\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062093 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062097    4827 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-token\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062107    4827 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062139    4827 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-workload-certs\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062148    4827 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdae895f-57fd-4f6c-8842-150ff9b37757-istiod-ca-cert\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062156    4827 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-istio-envoy\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062163    4827 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdae895f-57fd-4f6c-8842-150ff9b37757-credential-socket\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.062321 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.062172    4827 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9lcr\" (UniqueName: \"kubernetes.io/projected/cdae895f-57fd-4f6c-8842-150ff9b37757-kube-api-access-h9lcr\") on node \"ip-10-0-134-205.ec2.internal\" DevicePath \"\""
Oct 29 23:34:10.487065 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.487023    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" path="/var/lib/kubelet/pods/0aa6ad85-8294-49a5-8c24-7d4fb9e1582b/volumes"
Oct 29 23:34:10.487556 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.487476    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" path="/var/lib/kubelet/pods/59e63f57-51c7-426a-8967-3f8e1130f56d/volumes"
Oct 29 23:34:10.487902 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.487886    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7df262-3b46-4141-a339-bea7561b622f" path="/var/lib/kubelet/pods/ff7df262-3b46-4141-a339-bea7561b622f/volumes"
Oct 29 23:34:10.574776 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.574741    4827 generic.go:356] "Generic (PLEG): container finished" podID="cdae895f-57fd-4f6c-8842-150ff9b37757" containerID="76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333" exitCode=0
Oct 29 23:34:10.574963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.574831    4827 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"
Oct 29 23:34:10.574963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.574828    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5" event={"ID":"cdae895f-57fd-4f6c-8842-150ff9b37757","Type":"ContainerDied","Data":"76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333"}
Oct 29 23:34:10.574963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.574927    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5" event={"ID":"cdae895f-57fd-4f6c-8842-150ff9b37757","Type":"ContainerDied","Data":"83b6056f483daaea40f27f03b18131737821d26bf4cfe08a02dd355106397875"}
Oct 29 23:34:10.574963 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.574961    4827 scope.go:122] "RemoveContainer" containerID="76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333"
Oct 29 23:34:10.583953 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.583937    4827 scope.go:122] "RemoveContainer" containerID="76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333"
Oct 29 23:34:10.584262 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:34:10.584243    4827 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333\": container with ID starting with 76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333 not found: ID does not exist" containerID="76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333"
Oct 29 23:34:10.584330 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.584268    4827 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333"} err="failed to get container status \"76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333\": rpc error: code = NotFound desc = could not find container \"76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333\": container with ID starting with 76f34417653408e629b6c6e61cef32a1f4cda03666616725757c63657032a333 not found: ID does not exist"
Oct 29 23:34:10.623480 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.623453    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"]
Oct 29 23:34:10.634111 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:10.634084    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-r82j5"]
Oct 29 23:34:12.486622 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:12.486588    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdae895f-57fd-4f6c-8842-150ff9b37757" path="/var/lib/kubelet/pods/cdae895f-57fd-4f6c-8842-150ff9b37757/volumes"
Oct 29 23:34:13.028519 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:13.028455    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"]
Oct 29 23:34:13.038655 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:13.038631    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19522a-v4v9q"]
Oct 29 23:34:14.486346 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:14.486315    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a72d117-9954-4fbd-99b2-2854cffb4cb6" path="/var/lib/kubelet/pods/8a72d117-9954-4fbd-99b2-2854cffb4cb6/volumes"
Oct 29 23:34:19.703312 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:19.703278    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:34:20.293864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:20.293833    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:34:20.910711 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:20.910679    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:34:22.646413 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:22.646384    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6e461ae2-t89mx_d71c9cc0-c97c-4e58-b7e3-c7ab502c337e/revoke-keys/0.log"
Oct 29 23:34:23.030076 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:23.029994    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7ebcfc9e-9zd8v_dce60e1d-21f2-43a7-a6fa-436e5aea7cb0/revoke-keys/0.log"
Oct 29 23:34:23.209219 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:23.209191    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-937d9a20-jb9lf_09b13b1d-e33d-4399-bc4d-175bdafcee1c/revoke-keys/0.log"
Oct 29 23:34:23.775005 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:23.774981    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a0ed4fa4-8xfvr_2e87c67a-7b47-4aeb-8818-55d48813a7bc/revoke-keys/0.log"
Oct 29 23:34:23.957984 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:23.957959    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-afe349f3-b5dvl_475634d9-c3d3-4b50-b57e-7f3462c19aea/revoke-keys/0.log"
Oct 29 23:34:24.317715 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:24.317687    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b3c8d76d-mf7jx_616da088-a8a8-4779-8803-524aa43cebb2/revoke-keys/0.log"
Oct 29 23:34:24.490290 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:24.490266    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ba0a03df-b24f8_04413b7d-e0ce-42b6-874c-fc5152590b95/revoke-keys/0.log"
Oct 29 23:34:24.858846 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:24.858814    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d3a6107f-s9tx2_0d510b90-dedc-43b7-94da-1805c4dd9ff5/revoke-keys/0.log"
Oct 29 23:34:25.026467 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:25.026432    4827 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"]
Oct 29 23:34:25.036639 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:25.036609    4827 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9a20-jb9lf"]
Oct 29 23:34:25.394648 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:25.394620    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e7c70e40-dlncv_0858e0f5-5c00-43cd-8cd4-582d03797945/revoke-keys/0.log"
Oct 29 23:34:25.587285 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:25.587247    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f6b0afa6-58j7w_d9657ce0-51eb-46f9-a701-645f98c1b6f2/revoke-keys/0.log"
Oct 29 23:34:25.781171 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:25.781103    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff1b00de-4fz7f_41879f6b-c67d-48e6-88e6-7adca8234186/revoke-keys/0.log"
Oct 29 23:34:25.957746 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:25.957721    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4dzn6_e3ce4d59-d3fa-421f-979c-1fcc26069beb/postgres/0.log"
Oct 29 23:34:26.485412 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:26.485380    4827 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b13b1d-e33d-4399-bc4d-175bdafcee1c" path="/var/lib/kubelet/pods/09b13b1d-e33d-4399-bc4d-175bdafcee1c/volumes"
Oct 29 23:34:27.977570 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:27.977495    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cdfvp_9037b762-9013-436a-8abc-fc94d30b6f29/manager/0.log"
Oct 29 23:34:30.255701 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:30.255671    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d52czbg_b2d6d562-150a-4021-bf8b-153c01a7038d/istio-proxy/0.log"
Oct 29 23:34:30.450644 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:30.450617    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7wwjc_70eca953-9fa9-4c5b-b0ac-d88a23279013/discovery/0.log"
Oct 29 23:34:31.038877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:31.038849    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbc7f7f7f-k4vrd_636f1206-971e-4ef8-be31-8f2e6564589c/kube-auth-proxy/0.log"
Oct 29 23:34:32.335864 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:32.335836    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-x7gtt_d56dd7a0-03e2-4206-a490-555d93824666/istio-proxy/0.log"
Oct 29 23:34:32.821367 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:34:32.821332    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:34:32.893289 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:32.893260    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-tjghh_8160871b-24d7-4b6b-bbd5-64ed33ae6382/payload-processing/0.log"
Oct 29 23:34:34.304400 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:34.304367    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-c874f_89beb379-c7e3-441a-90cb-5572b7d934da/main/0.log"
Oct 29 23:34:34.670209 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:34.670180    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-external-token-limit-simulated-kserve-65fc98cd9c-mzd8x_f90fd47c-4d11-4faa-8eb9-d394960cc2cf/main/0.log"
Oct 29 23:34:35.071889 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:35.071813    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j67n8_d8d66d1c-4fef-4374-8b11-cf82ad34a314/main/0.log"
Oct 29 23:34:35.630132 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:35.630102    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6tqkb_fabbc053-4c55-47e4-8345-b2e049215d5d/main/0.log"
Oct 29 23:34:45.996378 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.996338    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9slr4/must-gather-r2wdz"]
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.996891    4827 cpu_manager.go:414] "containerMap: removing container" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy"
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.996930    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy"
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.996955    4827 cpu_manager.go:414] "containerMap: removing container" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy"
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.996986    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy"
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997000    4827 cpu_manager.go:414] "containerMap: removing container" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy"
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997007    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy"
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997023    4827 cpu_manager.go:414] "containerMap: removing container" podUID="cdae895f-57fd-4f6c-8842-150ff9b37757" containerName="istio-proxy"
Oct 29 23:34:45.997029 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997030    4827 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdae895f-57fd-4f6c-8842-150ff9b37757" containerName="istio-proxy"
Oct 29 23:34:45.997478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997132    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7df262-3b46-4141-a339-bea7561b622f" containerName="istio-proxy"
Oct 29 23:34:45.997478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997150    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e63f57-51c7-426a-8967-3f8e1130f56d" containerName="istio-proxy"
Oct 29 23:34:45.997478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997159    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa6ad85-8294-49a5-8c24-7d4fb9e1582b" containerName="istio-proxy"
Oct 29 23:34:45.997478 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:45.997168    4827 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdae895f-57fd-4f6c-8842-150ff9b37757" containerName="istio-proxy"
Oct 29 23:34:46.001611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.001590    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.018658 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.018624    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9slr4/must-gather-r2wdz"]
Oct 29 23:34:46.027419 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.027377    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9slr4\"/\"kube-root-ca.crt\""
Oct 29 23:34:46.027594 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.027461    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9slr4\"/\"openshift-service-ca.crt\""
Oct 29 23:34:46.028252 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.028232    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9slr4\"/\"default-dockercfg-spczd\""
Oct 29 23:34:46.062182 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.062144    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxcf9\" (UniqueName: \"kubernetes.io/projected/8e31ace1-4e89-4221-bb35-2eabc477f657-kube-api-access-sxcf9\") pod \"must-gather-r2wdz\" (UID: \"8e31ace1-4e89-4221-bb35-2eabc477f657\") " pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.062361 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.062287    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e31ace1-4e89-4221-bb35-2eabc477f657-must-gather-output\") pod \"must-gather-r2wdz\" (UID: \"8e31ace1-4e89-4221-bb35-2eabc477f657\") " pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.086450 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.086422    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rw5gl/must-gather-q72vx"]
Oct 29 23:34:46.089921 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.089905    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.109052 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.109030    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rw5gl/must-gather-q72vx"]
Oct 29 23:34:46.117965 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.117943    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rw5gl\"/\"openshift-service-ca.crt\""
Oct 29 23:34:46.118087 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.118027    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rw5gl\"/\"kube-root-ca.crt\""
Oct 29 23:34:46.118270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.118259    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rw5gl\"/\"default-dockercfg-r2dts\""
Oct 29 23:34:46.163692 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.163663    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f447ae92-4fdf-42cd-afa9-6493113283bc-must-gather-output\") pod \"must-gather-q72vx\" (UID: \"f447ae92-4fdf-42cd-afa9-6493113283bc\") " pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.163878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.163722    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e31ace1-4e89-4221-bb35-2eabc477f657-must-gather-output\") pod \"must-gather-r2wdz\" (UID: \"8e31ace1-4e89-4221-bb35-2eabc477f657\") " pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.163878 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.163816    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2fs\" (UniqueName: \"kubernetes.io/projected/f447ae92-4fdf-42cd-afa9-6493113283bc-kube-api-access-rz2fs\") pod \"must-gather-q72vx\" (UID: \"f447ae92-4fdf-42cd-afa9-6493113283bc\") " pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.163959 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.163908    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxcf9\" (UniqueName: \"kubernetes.io/projected/8e31ace1-4e89-4221-bb35-2eabc477f657-kube-api-access-sxcf9\") pod \"must-gather-r2wdz\" (UID: \"8e31ace1-4e89-4221-bb35-2eabc477f657\") " pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.164101 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.164085    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e31ace1-4e89-4221-bb35-2eabc477f657-must-gather-output\") pod \"must-gather-r2wdz\" (UID: \"8e31ace1-4e89-4221-bb35-2eabc477f657\") " pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.185325 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.185293    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfxq6/must-gather-wdmn5"]
Oct 29 23:34:46.185886 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.185864    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxcf9\" (UniqueName: \"kubernetes.io/projected/8e31ace1-4e89-4221-bb35-2eabc477f657-kube-api-access-sxcf9\") pod \"must-gather-r2wdz\" (UID: \"8e31ace1-4e89-4221-bb35-2eabc477f657\") " pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.188938 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.188920    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.207841 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.207820    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/must-gather-wdmn5"]
Oct 29 23:34:46.218270 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.218252    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfxq6\"/\"kube-root-ca.crt\""
Oct 29 23:34:46.218357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.218299    4827 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfxq6\"/\"openshift-service-ca.crt\""
Oct 29 23:34:46.218357 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.218326    4827 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfxq6\"/\"default-dockercfg-gxv8f\""
Oct 29 23:34:46.265091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.265024    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjftv\" (UniqueName: \"kubernetes.io/projected/76584916-4934-43b9-97f8-bf9e6f35d849-kube-api-access-vjftv\") pod \"must-gather-wdmn5\" (UID: \"76584916-4934-43b9-97f8-bf9e6f35d849\") " pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.265091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.265058    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76584916-4934-43b9-97f8-bf9e6f35d849-must-gather-output\") pod \"must-gather-wdmn5\" (UID: \"76584916-4934-43b9-97f8-bf9e6f35d849\") " pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.265091 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.265085    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2fs\" (UniqueName: \"kubernetes.io/projected/f447ae92-4fdf-42cd-afa9-6493113283bc-kube-api-access-rz2fs\") pod \"must-gather-q72vx\" (UID: \"f447ae92-4fdf-42cd-afa9-6493113283bc\") " pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.265329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.265250    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f447ae92-4fdf-42cd-afa9-6493113283bc-must-gather-output\") pod \"must-gather-q72vx\" (UID: \"f447ae92-4fdf-42cd-afa9-6493113283bc\") " pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.265560 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.265542    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f447ae92-4fdf-42cd-afa9-6493113283bc-must-gather-output\") pod \"must-gather-q72vx\" (UID: \"f447ae92-4fdf-42cd-afa9-6493113283bc\") " pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.287774 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.287750    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2fs\" (UniqueName: \"kubernetes.io/projected/f447ae92-4fdf-42cd-afa9-6493113283bc-kube-api-access-rz2fs\") pod \"must-gather-q72vx\" (UID: \"f447ae92-4fdf-42cd-afa9-6493113283bc\") " pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.314734 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.314709    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9slr4/must-gather-r2wdz"
Oct 29 23:34:46.367394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.366874    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjftv\" (UniqueName: \"kubernetes.io/projected/76584916-4934-43b9-97f8-bf9e6f35d849-kube-api-access-vjftv\") pod \"must-gather-wdmn5\" (UID: \"76584916-4934-43b9-97f8-bf9e6f35d849\") " pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.367394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.366925    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76584916-4934-43b9-97f8-bf9e6f35d849-must-gather-output\") pod \"must-gather-wdmn5\" (UID: \"76584916-4934-43b9-97f8-bf9e6f35d849\") " pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.367394 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.367353    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76584916-4934-43b9-97f8-bf9e6f35d849-must-gather-output\") pod \"must-gather-wdmn5\" (UID: \"76584916-4934-43b9-97f8-bf9e6f35d849\") " pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.388935 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.388901    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjftv\" (UniqueName: \"kubernetes.io/projected/76584916-4934-43b9-97f8-bf9e6f35d849-kube-api-access-vjftv\") pod \"must-gather-wdmn5\" (UID: \"76584916-4934-43b9-97f8-bf9e6f35d849\") " pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.403843 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.403818    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rw5gl/must-gather-q72vx"
Oct 29 23:34:46.504896 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.504867    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/must-gather-wdmn5"
Oct 29 23:34:46.612854 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.612824    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9slr4/must-gather-r2wdz"]
Oct 29 23:34:46.614512 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:34:46.614490    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e31ace1_4e89_4221_bb35_2eabc477f657.slice/crio-9983e4e3da817fc9f2c3a698559cc4cf249b88e7c488593ab58a13f8657dfe68 WatchSource:0}: Error finding container 9983e4e3da817fc9f2c3a698559cc4cf249b88e7c488593ab58a13f8657dfe68: Status 404 returned error can't find the container with id 9983e4e3da817fc9f2c3a698559cc4cf249b88e7c488593ab58a13f8657dfe68
Oct 29 23:34:46.616300 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.616284    4827 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:34:46.698014 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.697985    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rw5gl/must-gather-q72vx"]
Oct 29 23:34:46.699810 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:34:46.699783    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf447ae92_4fdf_42cd_afa9_6493113283bc.slice/crio-82ccfdc59186e55e566d2b5ed99337597f7dda3d1791e507b389e85fa1a7877c WatchSource:0}: Error finding container 82ccfdc59186e55e566d2b5ed99337597f7dda3d1791e507b389e85fa1a7877c: Status 404 returned error can't find the container with id 82ccfdc59186e55e566d2b5ed99337597f7dda3d1791e507b389e85fa1a7877c
Oct 29 23:34:46.761773 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.761743    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9slr4/must-gather-r2wdz" event={"ID":"8e31ace1-4e89-4221-bb35-2eabc477f657","Type":"ContainerStarted","Data":"9983e4e3da817fc9f2c3a698559cc4cf249b88e7c488593ab58a13f8657dfe68"}
Oct 29 23:34:46.762890 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.762864    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rw5gl/must-gather-q72vx" event={"ID":"f447ae92-4fdf-42cd-afa9-6493113283bc","Type":"ContainerStarted","Data":"82ccfdc59186e55e566d2b5ed99337597f7dda3d1791e507b389e85fa1a7877c"}
Oct 29 23:34:46.799726 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:46.799700    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/must-gather-wdmn5"]
Oct 29 23:34:46.801528 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:34:46.801503    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76584916_4934_43b9_97f8_bf9e6f35d849.slice/crio-de1554500ae51a463d3a29a84805a1e7aa909eec0ef26c200199b485606fe90b WatchSource:0}: Error finding container de1554500ae51a463d3a29a84805a1e7aa909eec0ef26c200199b485606fe90b: Status 404 returned error can't find the container with id de1554500ae51a463d3a29a84805a1e7aa909eec0ef26c200199b485606fe90b
Oct 29 23:34:47.825042 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:47.824982    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/must-gather-wdmn5" event={"ID":"76584916-4934-43b9-97f8-bf9e6f35d849","Type":"ContainerStarted","Data":"de1554500ae51a463d3a29a84805a1e7aa909eec0ef26c200199b485606fe90b"}
Oct 29 23:34:48.850880 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:48.850841    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/must-gather-wdmn5" event={"ID":"76584916-4934-43b9-97f8-bf9e6f35d849","Type":"ContainerStarted","Data":"8fe887de7758227aac2f98b35c589d8d54d21afff959e834cc75de335e44fb1c"}
Oct 29 23:34:49.861351 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:49.861314    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/must-gather-wdmn5" event={"ID":"76584916-4934-43b9-97f8-bf9e6f35d849","Type":"ContainerStarted","Data":"ccbce318b3a3129e9caeb0d13e19fe72b1403a217645e17dc05f56c9d77f422d"}
Oct 29 23:34:49.902032 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:49.901110    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zfxq6/must-gather-wdmn5" podStartSLOduration=2.413885235 podStartE2EDuration="3.901088148s" totalImagesPullingTime="1.487202913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:49.898505159 +0000 UTC m=+3125.521416178" watchObservedRunningTime="2026-10-29 23:34:49.901088148 +0000 UTC m=+3125.523999192"
Oct 29 23:34:51.704983 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:51.704944    4827 scope.go:122] "RemoveContainer" containerID="addaa7b867d8075bffdff0b0a14bb8c1217a34c6b2298becfc4fa7db191b6d69"
Oct 29 23:34:51.716083 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:51.716059    4827 scope.go:122] "RemoveContainer" containerID="10af68e42af0ae70706b176b664d870863326853c96096a096198cfb6ae51e16"
Oct 29 23:34:51.726920 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:51.726898    4827 scope.go:122] "RemoveContainer" containerID="9883ff9726cdfff36592e34f094135f7a0c849a4f82e21dac2797881940babd2"
Oct 29 23:34:51.741176 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:51.741145    4827 scope.go:122] "RemoveContainer" containerID="05250d6e9f5b065c89289ebcb55cfa1de92c3ae543d3d56574940a9eefa7dfc7"
Oct 29 23:34:51.751523 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:51.751494    4827 scope.go:122] "RemoveContainer" containerID="f38bef2f930b9e0701e06ba845fdd34f4b89988108625001a8d7a002c6f45a3b"
Oct 29 23:34:53.092244 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:53.092214    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nt7zn_34812142-44fe-4a32-b03e-97fc0a461bd5/global-pull-secret-syncer/0.log"
Oct 29 23:34:53.923080 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:53.922992    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hc2bd_c8ea7bf0-7270-447b-8aeb-a91164efe631/konnectivity-agent/0.log"
Oct 29 23:34:54.338867 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:34:54.335873    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-205.ec2.internal_cac70c0770f909e7c0598a461e924492/haproxy/0.log"
Oct 29 23:35:01.347477 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:01.347441    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cdfvp_9037b762-9013-436a-8abc-fc94d30b6f29/manager/0.log"
Oct 29 23:35:02.058951 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:02.056973    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rw5gl/must-gather-q72vx" event={"ID":"f447ae92-4fdf-42cd-afa9-6493113283bc","Type":"ContainerStarted","Data":"6ba833c26c3da63c0bea94fa53b26c0525aa404c33cb67429979473a879e4cc7"}
Oct 29 23:35:02.060195 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:02.059297    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9slr4/must-gather-r2wdz" event={"ID":"8e31ace1-4e89-4221-bb35-2eabc477f657","Type":"ContainerStarted","Data":"9ea86a8cc25c051135cf4811741a461260ae79e9f615e58808986d2b3ce90cd3"}
Oct 29 23:35:02.105404 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:02.105326    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9slr4/must-gather-r2wdz" podStartSLOduration=2.135050758 podStartE2EDuration="17.10530286s" totalImagesPullingTime="14.970252102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:02.103184483 +0000 UTC m=+3137.726095518" watchObservedRunningTime="2026-10-29 23:35:02.10530286 +0000 UTC m=+3137.728213903"
Oct 29 23:35:02.869682 ip-10-0-134-205 kubenswrapper[4827]: E1029 23:35:02.869616    4827 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7xwpt" containerName="extractor"
Oct 29 23:35:03.069286 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:03.069245    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rw5gl/must-gather-q72vx" event={"ID":"f447ae92-4fdf-42cd-afa9-6493113283bc","Type":"ContainerStarted","Data":"40bdfa8839dbe70e2200d4438c07422fdb40ce2b07192746d9acda4a48db966b"}
Oct 29 23:35:03.074019 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:03.073976    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9slr4/must-gather-r2wdz" event={"ID":"8e31ace1-4e89-4221-bb35-2eabc477f657","Type":"ContainerStarted","Data":"bbcc09c74fa5b2fa8ce6f6d6d2bf27eae5046d7d5a2d9f85400c30d21562d4c3"}
Oct 29 23:35:03.108913 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:03.108853    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rw5gl/must-gather-q72vx" podStartSLOduration=2.259902834 podStartE2EDuration="17.108833576s" totalImagesPullingTime="14.848930742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:03.107251758 +0000 UTC m=+3138.730162778" watchObservedRunningTime="2026-10-29 23:35:03.108833576 +0000 UTC m=+3138.731744596"
Oct 29 23:35:04.260402 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.259722    4827 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"]
Oct 29 23:35:04.273320 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.271039    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.285079 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.285052    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"]
Oct 29 23:35:04.459135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.455010    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-sys\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.459135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.455048    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmpq\" (UniqueName: \"kubernetes.io/projected/16c86ce9-6801-4e39-a28e-f1b9af78cd34-kube-api-access-dvmpq\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.459135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.455250    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-proc\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.459135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.455316    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-podres\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.459135 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.455396    4827 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-lib-modules\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.556839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556182    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-sys\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.556839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556240    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmpq\" (UniqueName: \"kubernetes.io/projected/16c86ce9-6801-4e39-a28e-f1b9af78cd34-kube-api-access-dvmpq\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.556839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556338    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-proc\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.556839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556368    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-podres\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.556839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556396    4827 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-lib-modules\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.556839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556543    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-lib-modules\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.556839 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556603    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-sys\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.557387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.556985    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-proc\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.557387 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.557050    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16c86ce9-6801-4e39-a28e-f1b9af78cd34-podres\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.581744 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.581442    4827 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmpq\" (UniqueName: \"kubernetes.io/projected/16c86ce9-6801-4e39-a28e-f1b9af78cd34-kube-api-access-dvmpq\") pod \"perf-node-gather-daemonset-6p7sj\" (UID: \"16c86ce9-6801-4e39-a28e-f1b9af78cd34\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.593233 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.593202    4827 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:04.764133 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.764096    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:35:04.910425 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.910397    4827 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"]
Oct 29 23:35:04.911461 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:04.911437    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:35:04.913676 ip-10-0-134-205 kubenswrapper[4827]: W1029 23:35:04.913652    4827 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod16c86ce9_6801_4e39_a28e_f1b9af78cd34.slice/crio-57d10ef8f4faad0da84f68c880e42f6fd950a5af928d278343055710f490e208 WatchSource:0}: Error finding container 57d10ef8f4faad0da84f68c880e42f6fd950a5af928d278343055710f490e208: Status 404 returned error can't find the container with id 57d10ef8f4faad0da84f68c880e42f6fd950a5af928d278343055710f490e208
Oct 29 23:35:05.100941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:05.092989    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj" event={"ID":"16c86ce9-6801-4e39-a28e-f1b9af78cd34","Type":"ContainerStarted","Data":"58cbd18a1c9db884755640e109fcc5ef2e4eefd17e23dd765e4dfd87d0fafb93"}
Oct 29 23:35:05.100941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:05.093068    4827 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj" event={"ID":"16c86ce9-6801-4e39-a28e-f1b9af78cd34","Type":"ContainerStarted","Data":"57d10ef8f4faad0da84f68c880e42f6fd950a5af928d278343055710f490e208"}
Oct 29 23:35:05.100941 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:05.094050    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:05.137491 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:05.137424    4827 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj" podStartSLOduration=1.137403987 podStartE2EDuration="1.137403987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:05.134618932 +0000 UTC m=+3140.757529943" watchObservedRunningTime="2026-10-29 23:35:05.137403987 +0000 UTC m=+3140.760315008"
Oct 29 23:35:06.132198 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:06.132157    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:35:06.285922 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:06.285849    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:35:06.695009 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:06.694969    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zspc8_be8bd6d3-88a1-482b-a514-d254705c5cbc/node-exporter/0.log"
Oct 29 23:35:06.809912 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:06.809886    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zspc8_be8bd6d3-88a1-482b-a514-d254705c5cbc/kube-rbac-proxy/0.log"
Oct 29 23:35:06.925362 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:06.925331    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zspc8_be8bd6d3-88a1-482b-a514-d254705c5cbc/init-textfile/0.log"
Oct 29 23:35:07.951824 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:07.951788    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:35:08.093552 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:08.093515    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:35:09.547899 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:09.547863    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:35:09.792317 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:09.792284    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:35:10.832542 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:10.832511    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:35:11.099898 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:11.099816    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:35:12.119877 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:12.119849    4827 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-6p7sj"
Oct 29 23:35:12.632423 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:12.632388    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:35:12.911799 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:12.911690    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:35:14.777748 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:14.777677    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:35:14.928674 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:14.928629    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-65lbj_3f894835-c3c1-4d46-a6c6-ca6d686676fe/manager/0.log"
Oct 29 23:35:16.047656 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:16.047626    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:35:16.696543 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:16.696509    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67b77b449f-756q9_0bcbd4d5-eb73-4d95-8d90-4f91a8379add/manager/0.log"
Oct 29 23:35:17.595393 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:17.595356    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:35:18.635787 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:18.633987    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqfnh_79884975-9bf8-48b4-89c1-95e4f33e8285/dns/0.log"
Oct 29 23:35:18.679138 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:18.679110    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d52czbg_b2d6d562-150a-4021-bf8b-153c01a7038d/istio-proxy/0.log"
Oct 29 23:35:18.733688 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:18.733657    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rkptj_e8b81688-d926-4ab2-a36d-5ec456980711/proxy/0.log"
Oct 29 23:35:18.747901 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:18.747874    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqfnh_79884975-9bf8-48b4-89c1-95e4f33e8285/kube-rbac-proxy/0.log"
Oct 29 23:35:19.386611 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:19.386574    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hg7lt_d4b0b131-a728-4cee-aa9e-efa80695c400/dns-node-resolver/0.log"
Oct 29 23:35:19.429062 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:19.429026    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d52czbg_b2d6d562-150a-4021-bf8b-153c01a7038d/istio-proxy/0.log"
Oct 29 23:35:20.199495 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:20.199462    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7wwjc_70eca953-9fa9-4c5b-b0ac-d88a23279013/discovery/0.log"
Oct 29 23:35:20.852955 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:20.852922    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d52czbg_b2d6d562-150a-4021-bf8b-153c01a7038d/istio-proxy/0.log"
Oct 29 23:35:20.913336 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:20.913305    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-76p4s_5612c1b5-6517-4a68-a0a4-3a10ff070d31/node-ca/0.log"
Oct 29 23:35:21.103990 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:21.103914    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7wwjc_70eca953-9fa9-4c5b-b0ac-d88a23279013/discovery/0.log"
Oct 29 23:35:22.195299 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:22.195261    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7wwjc_70eca953-9fa9-4c5b-b0ac-d88a23279013/discovery/0.log"
Oct 29 23:35:22.233698 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:22.233670    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbc7f7f7f-k4vrd_636f1206-971e-4ef8-be31-8f2e6564589c/kube-auth-proxy/0.log"
Oct 29 23:35:22.586567 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:22.586486    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-x7gtt_d56dd7a0-03e2-4206-a490-555d93824666/istio-proxy/0.log"
Oct 29 23:35:22.765621 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:22.765588    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-tjghh_8160871b-24d7-4b6b-bbd5-64ed33ae6382/payload-processing/0.log"
Oct 29 23:35:23.282526 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:23.282491    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbc7f7f7f-k4vrd_636f1206-971e-4ef8-be31-8f2e6564589c/kube-auth-proxy/0.log"
Oct 29 23:35:23.377156 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:23.377124    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d52czbg_b2d6d562-150a-4021-bf8b-153c01a7038d/istio-proxy/0.log"
Oct 29 23:35:23.681030 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:23.680990    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-x7gtt_d56dd7a0-03e2-4206-a490-555d93824666/istio-proxy/0.log"
Oct 29 23:35:23.884329 ip-10-0-134-205 kubenswrapper[4827]: I1029 23:35:23.884305    4827 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-tjghh_8160871b-24d7-4b6b-bbd5-64ed33ae6382/payload-processing/0.log"